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PREFACE 


the  Committee  on  Buman  Factors  was  established  In  October  1980  under 
the  joint  sponsorship  of  the  Office  of  Naval  Research  (ONR),  the  Air 
Force  Office  of  Scientific  Research  (AFOSR),  and  the  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  to  Identify 
basic  research  needs  of  the  military  services  In  support  of  human 
factors  engineering  applications  and  to  make  recommendations  for  basic 
research  that  will  Improve  the  foundations  of  this  discipline.  The 
committee's  first  meeting  was  held  In  December  1980;  in  October  1981 
the  National  Aeronautics  and  Space  Administration  (NASA)  joined  the 
sponsors  of  the  committee;  and  several  other  government  agencies  have 
expressed  Interest  In  the  committee's  work. 

Human  factors  Issues  arise  in  every  domain  In  which  humans 
interact  with  the  products  of  a  technological  society.  Consequently, 
the  knowledge  brought  to  bear  In  human  factors  applications  must  be 

drawn  from  a  vide  range  of  scientific  and  engineering  disciplines. 

✓ 

Although  no  small  group  can  be  fully  representative  of  all  disciplines 
relevant  to  human  factors,  the  expertise  represented  on  the  committee 
is  quite  broad.  It  includes  specialists  from  the  fields  of 
engineering,  biomechanic?,  psychology,  cognitive  science,  and 
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sociology  as  well  as  from  human  factors  engineering*  While  other 
disciplines  may  he  relevant.  It  Is  these  that  are  expected  to 
contribute  most  substantially  to  the  basic  data,  theory,  end  methods 
needed  to  improve  the  scientific  basis  of  human  factors. 

I  irlsh  to  thank  each  member  of  the  committee  for  their  thoughtful 
contributions  to  this  report.  Individual  members  or  small  groups  of 
members  accepted  primary  responsibility  for  authoring  each  chapter. 
This  authorship  is  acknowledged  in  the  note  at  the  beginning  of  each 
chapter.  All  committee  members,  whether  they  were  authors  or  not, 
deliberated,  reviewed,  and  contributed  to  Improvements  in  the  contean 
of  each  chapter.  I  am  especially  grateful  to  them  for  their  geperous 
contribution  of  time,  both  in  meetings  and  outside.  Their  efforts 
have  contributed  greatly  to  the  quality  of  this  report,  which  is  cmij 
a  product  of  the  full  committee.  Special  thanks  are  due  to  the  study 
director,  Robert  T.  Hennessy,  who  contributed  both  technically  and 
administratively  to  every  step  In  the  report's  development.  In 
addition,  he  has  taken  the  kind  of  Initiatives  that  made  It  possible 
for  me  to  chair  the  committee  with  minimum  effort  and  maximum  reward. 

Martin  A.  Tolcott  and  Gerald  S.  Malecki  of  the  Office  of  Naval 
Research,  Alfred  R.  Fregly  of  the  Air  Force  Office  of  Scientific 
Research,  Robert  M.  Sasmor  of  the  Army  Research  Institute,  and  Melvin 
D.  Montemerlo  of  the  National  Aeronautics  and  Space  Administration, 
representatives  of  the  committee's  sponsors,  have  also  made  important 
contributions.  Their  support,  encouragement,  and  Identification  of 
relevant  issues  have  been  most  helpful. 
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I  mo  grateful  also  to  the  participants  in  our  workshop  on  applied 
methods:  Stuart  K.  Card,  David  Keister,  Donald  L.  Parks,  Erich  P. 
Prlen,  and  John  B.  Shafer.  Their  broad  understanding  of  applied 
methods  and  their  cogent  appraisal  of  the  issues  and  needs  In  this 
area  formed  the  basis  for  Chapter  7  of  this  report. 

Several  people  were  helpful  to  the  committee  in  specific  ways. 

At  Wrlght-Patterson  Air  Force  Base,  Kenneth  R.  Boff  organized  a  series 
of  briefings  by  personnel  from  the  Air  Force  Aerospace  Medical 
Research  Laboratory  and  the  Air  Force  Human  Resources  Laboratory  as 
well  as  tours  of  several  of  their  research  facilities.  During  the 
committee's  visit  to  the  Naval  Training  Equipment  Center,  Walter  S. 
Chambers  and  Stanley  C.  Collyer  arranged  for  presentations  by  members 
of  the  Human  Factors  Laboratory  and  briefed  the  committee  on  the 
research  uses  of  the  visual  technology  research  simulator  as  well  as 
demonstrating  this  device.  I  extend  my  appreciation  to  these 
Individuals  and  organizations  for  their  efforts  on  the  committee's 
behalf . 

Many  other  individuals  also  hsve  contributed  to  the  work  of  the 
committee  and  thereby  to  the  contents  of  this  report.  A  number  of 
human  factors  professionals  provided  thoughtful  and  detailed  responses 
to  a  survey  on  research  Issues.  Others  served  as  outside  reviewers  of 
particular  chapters.  Karen  A.  English  and  M.  Jeanne  Richards  have 
served  ably  and  conscientiously  as  administrative  secretaries  over  the 
course  of  the  committee's  history.  Christine  L.  McShane,  editor  for 
the  Commission  on  Behavioral  and  Social  Sciences  and  Education, 
through  skill  and  perseverance  greatly  Improved  the  style  and  clarity 
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I 

INTRODUCTION  AND  OVERVIEW 


In  the  last  several  years  the  public  has  become  sensitized  to  the 
Importance  of  equipment  designed  to  accommodate  Its  human  users.  In 
the  course  of  events  at  the  Three  Mile  Island  nuclear  power  plant  many 
residents  of  Harrisburg  were  evacuated  because  of  the  accident 
precipitated  by  operators  misinterpreting  their  Instruments.  Coal 
miners  cover  equipment  lamps  intended  to  Illuminate  the  mine  wall, 
because  they  object  to  the  glare  in  their  faces.  The  M-l,  the  most 
technologically  sophisticated  battle  tank  ever  produced.  Is  limited  by 
the  operating  difficulties  experienced  by  its  crew.  With  computer 
terminals  now  pervasive  in  the  workplace,  more  users  are  voicing  their 
complaints  about  requirements  to  converse  in  arcane  dialects  of 
computer  languages. 

Each  of  these  examples  reflects  a  failure  to  consider  the  design 
of  a  system  from  the  point  of  view  of  its  potential  users;  thus  It  is 
not  surprising  that  the  public  is  demanding  that  more  attention  be 
paid  to  such  considerations.  These  demands  may  be  expressed  In  the 
decisions  of  jurors  In  court  cases  Involving  product  liability,  in  the 
renewed  emphasis  on  human  factors  In  military  and  aeronautics 

The  principal  author  of  this  chapter  is  Richard  W.  Pew. 
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laboratories,  and  In  the  Increase  in  job  opportunities  for  human 
factors  professionals  In  the  computer  industry.  In  March  1982,  over 
1,000  people  participated  In  a  conference  devoted  to  discussing  how  to 
make  computers  more  user-oriented. 

The  historical  roots  of  the  human  factors  profession  are  In 
Industrial  engineering  and  In  psychology.  In  the  early  1900s 
Frederick  W.  Taylor  coined  the  term  scientific  management,  by  which  he 
meant  the  application  of  scientific  principles  In  the  design  of  the 
Industrial  workplace.  Although  overzealous  "Taylorism"  resulted  In 
some  early  mismanagement,  his  work  formed  one  of  the  building  blocks 
for  modern  industrial  engineering  and  operations  research. 

During  the  latter  stages  of  World  War  II,  psychologists,  wr.c.  '..zi 
been  Involved  In  the  selection  and  training  of  aircraft  pilots,  were 
called  on  to  take  a  novel  perspective;.  Instead  of  selecting  pilots  to 
meet  the  severe  demands  of  the  cockpit,  they  were  asked  to  select  the 
cockpit  design  best  suited  to  the  characteristics  of  pilots.  This 
approach  reduced  accidents  and  allowed  a  larger  population  of 
potential  pilots  to  be  certified.  Because  flying  pushes  the  human 
body  to  its  physiological  limits,  the  effects  of  physiological  stress 
on  performance  became  a  further  consideration.  After  the  war  a  small 
group  of  universities  began  training  human  factors  specialists  for 
research  and  development  in  the  military  services  and  the  aerospace 
Industry. 

In  1957  the  Human  Factors  Society  was  formed  with  90  founding 
members;  by  1977  the  membership  had  grown  to  1,956;  and  in  the  last 
five  years  the  organization  has  expanded  by  an  additional  50  percent. 
In  addition,  various  engineering  societies  have  formed  groups  related 


to  human  factors.  The  formation  of  this  committee  within  the  National 
Research  Council  In  1980  is  the  latest  explicit  recognition  of  the 
importance  of  human  factors  in  today's  technological  society* 

Human  factors  engineering  can  be  defined  as  the  application  of 
scientific  principles,  methods,  and  data  drawn  from  a  variety  of 
disciplines  to  the  development  of  engineering  systems  in  which  people 
play  a  significant  role.  Successful  application  is  measured  by 
improved  productivity,  efficiency,  safety,  and  acceptance  of  the 
resultant  system  design.  The  disciplines  that  may  be  applied  to  a 
particular  problem  include  psychology,  cognitive  science,  physiology, 
biomechanics,  applied  physical  anthropology,  and  industrial  and 
systems  engineering.  The  systems  range  from  the  use  of  a  simple  tool 
by  a  consumer  to  multiperson  soclotechnical  systems.  They  typically 
include  both  technological  and  human  components. 

Human  factors  specialists  from  these  and  other  disciplines  are 
united  by  a  singular  perspective  on  the  system  design  process:  that 
design  begins  with  an  understanding  of  the  user's  role  in  overall 
system  performance  and  that  systems  exist  to  serve  their  users, 
whether  they  are  consumers,  system  operators,  production  workers,  or 
maintenance  crews.  This  user-oriented  design  philosophy  acknowledges 
human  variability  as  a  design  parameter.  The  resultant  designs 
incorporate  features  that  take  advantage  of  unique  human  capabilities 
as  well  as  build  in  safeguards  to  avoid  or  reduce  the  impact  of 
unpredictable  human  error. 

On  the  international  scene  this  collection  of  activities  has  been 
called  ergonomics,  meaning  the  study  of  work.  Its  practitioners  have 
placed  somewhat  more  emphasis  on  biomechanics  and  the  physiological 
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costs  of  doing  work  than  have  human  factors  practitioners  In  the 
United  States.  Aside  from  this  distinction,  the  two  terms  refer  to 
the  same ^collection  of  specialties. 

While  its  foundations  rest  ultimately  in  the  parent  disciplines , 
human  factors  research  focuses  on  the  solution  of  system  design 
problers  involving  more  than  one  of  these  disciplines.  Since  World 
War  II  the  major  sources  of  funding  for  basic  research  underlying 
human  factors  work  have  been  the  National  Aeronautics  and  Space 
Administration  (NASA)  and  the  military  services.  Since  the  passage  of 
the  Mansfield  Amendment  (Public  Law  91-441,  1970)  to  the  U.S.  defense 
budget,  which  mandated  a  shift  toward  system  development  and  away  x<.su 
basic  research,  the  real  dollar  volume  of  research  has  not  Increased: 
very  much.  What  research  there  Is  has  focused  increasingly  on 
short-term  goals.  As  a  result  the  basic  knowledge  needed  to  provide 
the  underpinnings  for  human  factors  applications  to  new  technology  has 
not  been  generated.  The  need  to  reverse  this  trend  is  at  least  part 
of  the  reason  that  the  military  services  and  NASA  have  taken  the 
Initiative  In  sponsoring  the  work  of  this  committee.  This  report 
reflects  the  committee's  recommendations  for  needed  research  In  terms 
of  both  long-term  end  short-term  objectives. 

This  report  does  not  attempt  to  cover  the  full  scope  of  human 
factors  engineering,  even  In  relation  to  military  and  NASA  needs.  As 
ths  committee  began  discussing  research  needs,  a  wide  range  of 
possible  topics  was  considered.  Two  of  our  meetings  included  tours 
and  discussions  of  ongoing  research  In  military  laboratories. 

Committee  members  were  encouraged  to  develop  brief  position  papers  on 
highlighted  topics  that  were  germane  to  their  interests.  The  human 
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factor*  community  was  surveyed  through  an  article  In  the  Bulletin  of 
the  Human  Factor*  Society,  and  116  response*  were  received;  the  survey 
result*  confirmed  the  importance  of  many  of  the  topic*  already 
identified  by  the  committee.  Some  topic*  were  dropped,  and  some  new 
paper*  were  generated.  Others  were  combined  into  coherent  units; 
still  others  were  deferred  for  further  study  or  initiative. 

The  material  in  this  report  is  the  result  of  that  process.  Each 
chapter  is  designed  to  be  a  self-contained  report  of  an  important  area 
in  which  research  is  needed.  All  the  topics  discussed  here  meet  the 
following  criteria:  (1)  each  topic  is  germane  to  our  military  and 
NASA  sponsors;  (2)  the  topics  sre  within  the  expertise  of  the 
committee;  (3)  each  topic  has  been,  in  the  opinion  of  the  committee, 
Incompletely  addressed  by  previous  or  current  military  and  civilian 
research  efforts;  and  (4)  the  potential  results  of  the  recommended 
research  will  be  Important  contributions  to  the  scientific  basis  and 
practice  of  human  factors.  And  the  work  of  the  committee  is  ongoing. 
In  addition  to  the  research  areas  presented  in  this  report,  work  on  a 
number  of  topics  is  in  various  stages  of  development:  (1) 
organizational  context  in  relation  to  design;  (2)  team  performance; 

(3)  simulation;  (4)  human  performance  modeling;  (5)  multicolor 
displays;  (6)  human  factors  education,  and  (7)  accident  reporting 
systems.  We  expect  to  address  many  of  these  as  well  as  other  topics 
in  subsequent  reports. 

In  the  paragraphs  that  follow,  the  areas  of  research  suggested  by 
the  ccimsittee  are  summarized  together  with  some  of  our  major 
recommendations.  The  chapters  themselves  provide  a  detailed 
elaboration  of  these  topics. 
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HUMAN  DECISION  MAKING 


A  central  issue  in  the  understanding  of  human  performance  Is  human 
decision  making*  It  has  become  even  more  important  with  the  Increased 
role  of  automation  in  complex  modern  systems  ranging  from  military 
command,  coutrol,  and  communication  systems  to  aircraft  and  process 
control  systems.  There  has  been  much  support  for  research  on  decision 
making  over  the  last  IS  years,  particularly  by  the  Defense  Advanced 
Research  Projects  Agency  and  the  Office  fox  Naval  Research.  This 
research  has  tended  to  focus  on  formal  decision  theoretic 
constructions,  which,  while  analytically  powerful,  have  proved  to  be 
insufficiently  robust  to  reflect  the  strengths  and  weakneaoes  of  human 
decision-making  capabilities.  Tha  committee  recommends  further 
research,  with  en  emphaele  on  moving  into  uncharted  areas. 

Surprisingly,  despite  the  effort  devoted  to  decision-making 
research  in  general,  there  la  still  a  need  for  research  on  how  to 
structure  practical  decision  problems  end  on  improving  the  realism  of 
models  that  claim  to  relate  to  decision-making  performance.  He  do  not 
know  how  to  represent  decision  situations  that  evolve  dynamically,  nor 
do  we  have  a  systematic  framework  from  which  to  consider  decision 
aiding. 

Furthermore,  we  ere  coming  tc  realise  that  many  planning 
activities  actually  Involve  decision  making  that  cannot  be  modeled  by 
enumerating  the  possible  states  of  the  world  and  courses  of  action  in 
a  unitary  decision  matrix.  They  often  evolve  o vat  time  in  bits  and 
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pieces  with  limited  central  direction*  We  need  a  deeper  undere tending 
of  much  diffuse  decision  processes  in  order  to  provide  effective 
computer  aids  for  this  kind  of  decision. 

While  previous  work  has  led  to  many  decision-making  aids  and 
models,  no  criteria  or  methodologies  have  been  suggested  for 
evaluating  their  relative  merits*  Until  such  comparisons  are  made, 
practitioners  will  continue  to  advocate  their  own  products  without  a 
basis  for  choice  among  them*  Finally,  there  is  a  persistent  need  for 
development  of  innovative  ways  of  soliciting  preference  and  relative 
value  judgments  from  people,  a  problem  that  leads  us  directly  to  the 
second  topic. 

ELICITING  EXPERT  JUDGMENT 

The  application  of  expert  judgment  covers  everything  from  medical 
evaluations  to  accident  investigations*  Although  the  subject  matter 
ranges  widely,  it  is  our  belief  that  there  are  generic,  substantive 
research  issues  that  should  be  addressed  In  a  coherent  program.  These 
problems  recur  in  diverse  contexts  for  which  elicitation  methods 
either  do  not  exist  or  are  Inadequately  standardized  across 
applications  to  yield  consistent  results.  The  research  issues  include 
(1)  creating  a  common  frame  of  reference  from  which  to  assess 
judgments  among  a  group  of  experts;  (2)  formulating  questions  for 
experts  in  a  way  that  is  compatible  with  their  mental  structures  or 
cognitive  representations  of  a  problem;  (3)  eliciting  judgments  about 
the  quality  of  lnfotmatlon;  (4)  detecting  and  identifying  reporting 
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bias  In  Judgments;  and  (5)  minimising  the  effects  of  memory  loss  and 
distortion  on  the  reporting  of  past  events. 

SUPERVISORY  CONTROL  SYSTEMS 

Supervisory  control  Is  a  relatively  new  conceptualisation  of  system 
function  that  Is  playing  an  Increasingly  Important  role  In  automated 
systems.  In  such  systems,  operators  supervise  the  semiautomatic 
control  of  a  dynamic  process,  such  as  a  chemical  plant  or  railway 
system.  Typically  the  operators  work  In  teams  and  control  computed, 
which  in  turn  mediate  Information  flow  among  various  automatic 
components.  Other  examples  of  supervisory  control  systems  are  modern 
aircraft,  medical  Intensive  care  units,  power  plants,  and  distributed 
command  and  control  systems  such  as  may  be  found  In  military 
operations  or  In  manufacturing  by  robots.  Such  systems  deemphasize 
the  importance  of  human  sensory  and  motor  capabilities  and  emphasize 
complex  perceptual  and  cognitive  skills.  This  perspective  Is 
relatively  new  to  practicing  system  designers;  work  Is  beginning  to  be 
sponsored  In  these  areas,  but  much  further  development  Is  needed. 

Supervisory  control  may  be  thought  of  as  a  generalization  from 
earlier  work  on  monitoring  and  controlling  complex  systems;  in  that 
sense  the  foundations  for  modeling  and  theory  are  established.  The 
theory  must  be  greatly  elaborated  and  extended,  however,  to  meet  the 
analysis  requirements  of  current  and  future  systems.  As  the  human 
skills  of  thinking,  reasoning,  planning,  and  decision  making  become 
key,  the  models  must  be  able  to  accommodate  these  human  capacities  and 
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limitations.  This  is  s  choice  opportunity  to  bring  together  work  on 
control  theory  models  end  cognitive  science  representations. 

Cognitive  psychology  is  also  advancing  our  understanding  of  the 
way  in  which  resources  are  shared  among  various  processes  within  the 
brain*  This  work  has  unexplored  implications  for  understanding  how  to 
modify  system  design  to  change  perceived  workload,  particularly  in  the 
complex  tasks  typical  of  supervisory  control.  Each  of  the  military 
services  has  research  programs  focused  on  human  workload  analysis.  In 
our  opinion  many  of  them  are  too  application-oriented;  they  need  a 
stronger  focus  on  research  to  advance  the  knowledge  base  from  which 
new  application  techniques  will  emerge. 

Another  key  concern  In  supervisory  control  is  prediction  and  the 
control  of  human  error.  Our  understanding  of  this  topic  is  in  its 
Infancy.  We  have  no  general  theory  of  human  error,  although  theories 
abound  for  human  response  time.  Human  reliability  analysis  has  been 
in  vogue  for  several  years,  but,  as  currently  practiced,  it  simply 
uses  the  numerical  aggregation  of  historical  data  on  recorded  human 
failure  rates.  It  is  weakest  in  just  the  situations  in  which  it  is 
most  needed — when  the  activity  Involves  complex  diagnosis,  situation 
assessment,  and  interaction  with  computers. 

At  the  level  of  design,  there  are  three  major  questions:  how  to 
design  supervisory  control  tasks  to  accommodate  human  capabilities  and 
limitations;  how  to  organize  and  display  the  information  needed  to 
carry  out  these  tasks;  and  how  much  control  to  delegate  to  the  human 
versus  the  automatic  parts  of  the  system. 


USER-COMPUTER  INTERACTION 


Since  computers  ere  already  playing  a  major  role  in  most  new  system 
developments,  Including  supervisory  control  systems,  issues  of 
facilitating  the  learning  and  use  by  both  computer  professionals  and 
novices  has  been  accorded  a  chapter  of  its  own* 

At  a  March  1982  conference  on  user-computer  interaction,  more 
than  100  papers  addressed  a  variety  of  topics  related  to  hardware  and 
software  design.  More  than  half  of  the  1,000  participants  were  system 
design  specialists  from  Industry  and  government.  The  committee 
believes  that  this  level  of  interest  foretells  a  heavy  demand  for 
scientific  knowledge  that  has  yet  to  be  created.  Although  a  number  of 
industrial  laboratories  are  supporting  proprietary  work,  there  is  only 
one  major  funded  collaborative  effort  between  computer  science  and 
human  factors  specialists,  that  at  Virginia  Polytechnic  Institute  and 
State  University  (funded  by  the  Office  for  Naval  Research). 

Moat  human  factors  research  has  been  done  in  the  area  of  computer 
hardware.  Information  is  available  on  which  to  base  design  decisions 
concerning  information  display  hardware  and  keyboards.  Many 
alternative  input  devices,  such  as  Joy  sticks,  track  balls,  and  light 
pens,  have  been  studied  in  the  context  of  specific  applications. 

There  is  a  need  for  further  work  on  input  devices  thst  focuses  on 
comparison  among  the  full  range  of  devices  across  a  broad  set  of  uses, 
including  instruction,  text  processing,  and  graphics. 
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Automatic  speech  recognition  end  production  have  attracted  much 
Interest  as  the  technology  Improves*  Speech  as  an  alternative  to 
manual  and  visual  modes  of  Input  and  output  needs  systematic 
Investigation.  Fundamental  work  Is  necessary  on  the  design  of 
Interactive  speech  dialogs  that  Involve  inherently  sequential 
communication  and  potentially  heavy  memory  demands  on  the  listener. 

As  computer  terminals  are  becoming  pervasive  In  the  white-collar 
workplace,  concern  Is  growing  about  the  adverse  effects  on  people  from 
long-term  use  of  terminals  with  cathode  ray  tubes  (CRTs).  A  recent 
study  by  the  National  Institute  for  Occupational  Safety  and  Health 
found  no  radiation  hazards  from  CRTs  but  did  find  a  substantial 
Increase  in  worker  complaints  of  fatigue  and  other  health  problems 
from  sustained  dally  use.  This  study  was  not  able  to  distinguish  CRT 
design-based  complaints  from  those  relating  to  the  task  or  other 
features  of  the  workplace— and  this  Is  an  urgent  research  need.  In 
Europe,  governments  are  now  mandating  standards  for  workplace 
designs.  It  will  not  be  long  before  similar  actions  are  taken  In  the 
United  States  and  the  research  must  begin  now  to  anticipate  them. 

In  the  area  of  software  design,  research  needs  are  only  beginning 
to  be  filled.  Effective  design  of  sophisticated  software  implies 
understanding  of  human  knowledge  sytems  and  the  ability  to  represent 
not  only  what  a  user  knows  but  also  how  a  user  makes  inferences  from 
that  Information.  There  Is  a  need  for  models  of  users'  understanding 
of  the  system  with  which  they  are  Interacting,  a  problem  that  is 
Important  for  supervisory  control  applications  as  well. 

Perhaps  the  most  neglected  research  area  in  computer  system 
development  Is  how  to  produce  effective  materials  and  reference 
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Information*  While  design  principles  developed  for  printed  materials 
are  useful  for  computer  system  documentation,  there  are  documentation 
opportunities  unique  to  Interactive  systems  that  we  do  not  yet 
understand  how  to  exploit  effectively. 

Finally,  there  is  a  need  to  understand  in  more  detail  the 
characteristics  of  the  user  population  that  make  a  difference  in 
computer  system  design.  We  need  research  that  suggests,  in  parametric 
terms,  how  changes  in  user  characteristics  should  be  reflected  in 
system  design  changes. 

The  committee  regards  user-computer  Interaction  as  one  of  the 
most  urgent  topics  on  which  to  undertake  research  initiatives. 


POPULATION  GROUP  DIFFERENCES 


Through  public  sentiment  as  well  as  government  legislation,  our 
society  has  mandated  the  elimination  of  discrimination  among 
population  groups  in  the  design  of  jobs  and  workplaces.  In  addition 
to  racial  discrimination,  there  is  growing  concern  about 
discrimination  on  the  basis  of  sex,  age,  and  disability.  We  lack  the 
research  necessary  to  describe  the  nature  and  extent  of  performance 
differences  among  the  various  population  groups  about  which 
discrimination  is  a  concern.  The  committee  believes  it  is  in  the 
national  interest  to  undertake  the  research  necessary  to  accommodate 
this  relatlonshp  between  population  group  differences  and  design. 

It  is  not  enough  to  consider  population  group  differences  per 
se.  In  some  cases  the  effect  of  a  group  characteristic  such  as  age  on 


performance  may  depend  on  the  value  of  some  other  variables,  such  as 
amount  of  training  or  level  of  Interpersonal  skills.  It  may  be 
misleading  to  discover  simply  that  performance  deteriorates  with  age, 
when  In  fact  training  or  experience  may  reverse  that  trend.  Such 
Interactions  remain  largely  unexplored. 

There  Is  also  a  need  to  understand  the  way  in  which  theae 
differences  In  performance  should  influence  workplace  design  or 
training  procedures.  We  know  how  to  write  equipment  specifications 
designed  to  fit  95  percent  of  a  particular  user  population  insofar  a6 
body  dimensions  are  concerned,  but  for  most  other  human  performance 
characteristics  we  lack  this  knowledge. 

APPLIED  METHODS 

Much  human  factors  work  la  performed  under  constraints  of  money,  time, 
and  opportunity  that  preclude  the  use  of  the  kind  of  experimental 
methods  used  in  laboratory  research.  From  necessity,  human  factors 
practitioners  have  adopted  or  developed  a  variety  of  applied  methods 
for  acquiring  or  organizing  Information  related  to  human 
characteristics  that  arise  In  the  context  of  system  design, 
development,  and  evaluation.  Examples  of  these  methods  are  task 
analysis.  Information  flow  analysis,  collection  and  analysis  of  survey 
data,  evaluation  of  physical  mock-ups,  and  the  structured  walk 
through.  In  contrast  to  the  methods  of  scientific  research,  which  are 
maintained  and  disseminated  In  university  curricula  and  textbooks,  and 
by  specialists  who  devote  careers  to  Improving  and  inventing 
experimental  design  procedures,  applied  methods  In  human  factors  work 
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are  described  only  briefly  In  technical  project  reports,  which  are 
difficult  to  access,  and  efforts  to  Improve  or  invent  methods  occur 
largely  In  connection  with  a  particular  project* 

There  Is  a  clear  need  to  develop  a  compendium  of  standard 
descriptions  of  the  most  Important  applied  methods.  This  compendium 
would  be  valuable  for  use  in  human  factors  curricula  in  colleges  and 
universities  and  for  continuing  education  tutorials  for  human  factors 
practitioners.  Currently  most  knowledge  of  applied  methods  Is  gained 
through  on-the-job  experience. 

Documenting  existing  applied  methods,  however,  will  not  full  til 
the  methodological  needs  for  all  current  and  future  system  design 
purposes.  Advances  In  computer  technology  applied  to  automatl&u  «r*c. 
supervisory  control  systems  and  computer  systems  themselves  all  have 
profound  methodological  Implications  for  the  analysis  and  description 
of  the  roles  people  play  In  these  sytems.  Existing  methods  such  as 
workload  analysis,  protocol  analysis,  and  function  allocation  require 
research  to  modify  and  extend  their  use  In  new  applications  In  which 
the  emphasis  Is  on  cognitive  functions  of  operators  rather  than  on  the 
perceptual-motor  functions  prominent  in  old  systems. 

Similarly,  there  Is  a  need  to  develop  new  methods  to  provide 
Information  of  the  type  and  form  necessary  to  resolve  such  Issues  as 
translating  task  requirements  Into  personnel  selection  criteria, 
deriving  training  requirements  from  functional  requirements,  and 
describing  or  evaluating  the  effects  of  task  or  system  functions  on 
the  affective  responses  of  personnel. 

All  the  basic  research  needs  addressed  In  this  report  require 
experimental  investigations  to  provide  the  theory,  principles,  and 
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data  to  support  human  factors  work  In  the  design  and  evaluation  of 
systems.  The  application  of  the  knowledge  derived  from  basic 
research,  however,  will  occur  largely  through  the  use  of  applied 
methods.  Documentation  of  existing  methods  and  research  to  extend  and 
Initiate  methods  to  meet  future  needs  are  as  essential  as  the 
substantive  research  to  Improve  both  the  scientific  basis  and  the 
practical  effectiveness  of  human  factors  work. 


CONCLUSION 


System  design  and  the  world  of  work  are  undergoing  profound  changes. 

In  a  period  when  automation  Is  replacing  the  need  for  finely  tuned 
perceptual-motor  activities  by  skilled  operators,  human  productivity 
Is  no  longer  easily  assessed  In  terms  of  unit  output.  New  systems 
place  increased  demands  on  the  cognitive  and  decision-making  aspects 
of  human  performance.  The  role  of  people  in  systems  is  shifting  to 
those  of  monitoring  and  directing  otherwise  automatic  processes  in 
industrial  production,  transportation,  military  operations,  and  office 
work. 

These  changes  in  human-machine  relations  both  offer  new 
opportunities  and  present  nev  problems  for  system  design.  It  is 
therefore  timely  and  appropriate  that  the  committee's  first  report  of 
research  needs  in  human  factors  emphasizes  the  Importance  of 
understanding  fundamental  cognitive  processes  and  their  role  in 
Interactive  and  supervisory  control  systems. 
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HUMAN  DECISION  MAKING 


Work  organizations,  and  those  who  staff  them,  rise  and  fall  by  their 
ability  to  make  decisions.  These  may  be  major  strategic  decisions,  such 
as  the  deployment  of  forces  or  inventories,  or  local  tactical  decisions, 
such  as  how  to  promote,  motivate,  and  understand  particular 
subordinates.  To  list  the  klnd3  of  decisions  that  need  to  be  made  and 
the  stakes  that  sometimes  ride  on  them  would  be  to  repeat  the  obvious. 
Decisions  are  made  explicitly  whenever  one  consciously  combines  beliefs 
and  values  in  order  to  choose  a  course  of  action.  They  are  made 
implicitly  whenever  one  relies  on  a  ritualized  response  (habit, 
tradition)  to  cope  with  a  choice  between  options.  Repetition  of  past 
decisions  may  result  in  suboptimal  choices;  however,  It  may  also  provide 
a  ready  escape  from  the  difficulties  and  expense  of  explicit  decision 
making.  The  reasons  decision  making  often  seems  (and  is)  so  difficult 
are  quite  varied,  as  are  the  opportunities  for  interventions  and  the 
needs  for  human  factors  research  to  buttress  those  interventions. 

One  problem  is  Information  overload:  More  things  need  to  be 
considered  than  can  be  held  and  manipulated  in  one's  head 
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simultaneously.  Coping  with  such  computational  problems  lc  an  ideal  task 
for  computers,  and  there  are  a  variety  of  software  packages  available 
that  in  one  way  or  another  combine  decision  makers'  beliefs  and  values  in 
order  to  produce  a  recommendation.  Choosing  between  and  using  these 
decision  aids  forces  one  to  face  a  second  inherent  difficulty  of  decision 
making:  not  knowing  how  to  define  (or  structure)  the  decision  problem 
and  to  assess  one's  own  values,  that  is,  how  to  make  trade-offs  between 
competing  objectives.  Because  analytic  decision-making  methods  cannot 
operate  without  guidance  on  these  issues,  judgment  is  an  inevitable  part 
of  the  decision-making  process,  as  is  the  need  for  judgment  elicitation 
methods  to  complement  the  decision  aid  (see  Chapter  3).  A  third 
difficulty  is  knowing  when  to  atop  analyzing  and  start  acting.  Taking 
that  atep  requires  one  to  assess  the  quality  of  the  decision-making 
process  and  reconcile  any  remaining  conflicts  between  the  recommendation 
it  produces  and  that  produced  by  one's  own  intuitions.  To  help  one 
through  this  step,  a  decision  aid  must  reveal  its  own  limits  in  ways  that 
are  psychologically  meaningful.  A  fourth  difficulty  is  that  in  many 
interesting  decisions  one  knows  too  little  to  act  confidently.  When 
uncertainty  is  a  fact  of  life,  the  role  of  good  design  is  to  ensure  that 
the  best  use  is  made  of  all  that  is  known. 

The  existence  of  these  four  problems  is  common  knowledge.  Their 
resolution  is  complicated  by  a  fifth  difficulty  whose  Identification 
requires  research:  People's  commonsense  judgments  are  subject  to  robust 
and  systematic  biases.  These  biases  make  it  difficult  to  rely  on 
Intuition  as  a  criterion  for  the  adequacy  of  decisions  and  the  methods 
that  produce  them.  Decision  aids  must  accommodate  these  biases  and  may 
require  supplementary  training  exercises  lest  their  recommendations  be 
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adopted  only  when  they  affirm  Intuition*  that  are  known  to  be  faulty. 

Given  the  multitude  of  decisions  that  are  made,  any  research  or 
design  effort  that  made  even  a  minute  contribution  to  the  quality  of  a 
minute  proportion  of  all  decisions  would  bear  a  large  benefit  in  absolute 
terms.  Proving  that  such  a  benefit  had  been  derived  would  be  as 
difficult  as  it  is  in  most  areas  of  human  factors  work.  Whenever 
uncertainty  is  involved,  better  decisions  will  produce  outcomes  only  over 
the  long  run.  That  makes  it  difficult  to  establish  the  validity  of  bona 
fide  Improvements  and  easy  to  fall  prey  to  highly  touted  methods  with 
good  face  validity,  but  little  else.  A  sound  research  base  is  needed  not 
only  to  develop  better  decision-making  methods,  but  also  to  give  users  a 
fighting  chance  at  being  able  to  identify  which  methods  are  Indeed  better 
for  their  purposes. 


BACKGROUND 


Ad  hoc  advice  to  decision  makers  can  be  traced  from  antiquity  to  the 
Sunday  supplements.  Scientific  study  of  decision  making  probably  begins 
with  the  development  of  statistical  or  Bayesian  decision  theory  by  Borel, 
Ramsey,  de  Flnettl,  von  Neumann,  Morgenctern,  Venn,  Wald,  and  others. 
They  showed  how  to  characterise  and  Interrelate  the  primitives  of  a 
general  model  of  decision-making  situations,  highlighting  its  subjective 
elements.  The  development  of  scientific  decision  aids  could  be  traced  in 
the  work  of  Edwards,  Ralffa,  Schlaifer,  and  others,  who  showed  how 
complex  reel-world  decision  situations  could  be  Interpreted  in  terms  of 
the  general  model.  Essential  to  this  model  is  the  notion  that 
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decision-making  problems  -an  be  decomposed  Into  components  that  can  be 
assessed  individually,  then  combined  into  a  general  recommendation  that 
reflects  the  decision  makers'  best  interest*  Those  components  are 
typically  described  as  options,  beliefs,  and  values  or  alternatives, 
opinions,  snd  preferences,  or  some  equivalent  triplet  of  terns.  They  are 
Interrelated  by  an  integration  scheme  called  a  decision  rule  or  problem 
structure  (e.g.,  Flschhoff,  at  ax.,  1981;  Sage,  1981). 

More  generally,  decision-making  models  typically  envision  four 
Interrelated  steps. 

1.  Identify  all  relevant  courses  of  action  among  which  the  decision 
maker  may  choose.  This  choice  among  options  (or  alternatives) 
constitutes  the  act  of  decision;  the  deliberations  that  precede  it  are 
considered  to  be  part  of  the  decision-making  process. 

2.  Identify  the  consequences  (advantages)  that  may  arise  as  a  result 
of  choosing  each  option;  assess  their  relative  attractiveness.  In  this 
act  the  decision  maker's  values  find  their  expression.  Although  these 
values  are  essentially  personal,  they  may  be  clarified  by  techniques  such 
as  multiattribute  utility  analysis  and  informed  by  economic  techniques 
that  attempt  to  establish  the  market  value  of  consequences. 

3.  Assess  the  likelihood  of  these  consequences'  being  realized. 

These  probabilities  may  be  elicited  by  straightforward  judgmental  methods 
or  with  the  aid  of  more  sophisticated  techniques,  such  as  fault  tree  and 
event  tree  analysis.  If  the  decision  maker  knows  exactly  what  will 
happen  given  each  course  of  action,  it  then  becomes  a  case  of  decision 
making  under  conditions  of  certainty  and  this  stage  drops  out. 

4.  Integrate  all  these  considerations  in  order  to  identify  what 
appears  to  be  the  best  option.  Making  the  best  of  what  is  or  could  be 


5 


known  at  the  time  of  the  decision  1*  the  halloark  of  good  decision 
making.  The  decision  maker  la  not  to  be  held  responsible  If  this  action 
meets  with  misfortune  and  an  undesired  option  Is  obtained. 

These  steps  are  both  demanding  and  vague.  Fulfilling  them  requires 
considerable  attention  to  detail  and  may  be  accomplished  In  a  variety  of 
ways.  Moreover,  they  may  not  even  be  followed  sequentially.  If  Insights 
gained  at  one  step  lead  the  decision  maker  to  revise  the  analysis 
performed  at  a  different  step.  This  flexibility  has  produced  a  variety 
of  models  and  methods  of  decision  making  whose  Interrelations  are  not 
always  clearly  specified. 

The  opportunity  for  routlnlzlng  and  merchandising  these 
decision-making  procedures  led  to  one  of  the  academic  and  consulting 
growth  Industries  of  the  1970s.  A  wide  variety  of  software  packages  and 
firms  can  now  bring  the  fruits  of  these  theoretical  advances  to 
practicing  decision  makers.  Decision  analysis,  the  most  common  name  for 
these  procedures,  is  part  of  the  curriculum  of  most  business  schools. 
Although  It  has  met  considerable  Initial  resistance  from  decision  makers 
because  of  its  novelty  and  because  of  the  explicitness  about  values  and 
beliefs  that  It  requires,  decision  analysis  seems  to  be  gaining 
considerable  acceptance  (e.g.,  Bonczek,  et  el.,  1981;  Brown,  et  al., 

1974;  Ralffa,  1968).  This  acceptance  seems,  even  now,  to  go  beyond  what 
could  be  justified  on  the  basis  of  any  empirical  evidence  of  its 
efficacy.  Figure  2-1  gives  some  examples  of  the  contexts  within  which 
decision-siding  schemes  relying  on  Interactive  computer  eysteas  have  been 
operating  and  have  been  reported  In  the  professional  literature.  Figure 
2-2  ie  similar  to  the  summary  printout  of  one  such  seneme,  which  offers 
physicians  on-line  diagnoses  of  the  causes  of  dyspepsia. 
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Accounting— helping  to  assess  the  financial  viability  of  corporations. 

Clinical  diagnosis — helping  physicians  to  decide  whether  to  perform 
diagnostic  procedures  and  how  to  Interpret  their  results. 

Counseling— helping  people  to  choose  careers  or  consider  having  children. 

Energy— choosing  where  to  site  energy-producing  facilities. 

Meteorology— derivation  of  precipitation  forecasts. 

Military— deciding  whether  troops  are  in  an  adequate  state  of 
readiness;  preplanning  responses. 

Petroleum  geology— allocation  of  resources  for  oil  exploration. 

Pharmaceutics — helping  in  monitoring  field  reports  in  order  to  decide 
whether  drugs  need  to  be  recalled. 

Pesearch  and  development — deciding  how  to  allocate  funds. 


FIGURE  2-1  Examples  of  Operating  Decision-Aiding  Systems 
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ROTHERMAN  AREA  HEALTH  AUTHORITY 
MONTAGU  HOSPITAL 
SYMPTOM  PROCESSING  PROJECT 


UNIT  NO.  1  456/89 
SURNAME:  Smith 
FIRST  NAMES:  John 


HISTORY  SHEET 


CLINICIAN:  Dr.  Gardner 


SYMPTOMS  INPUT  TO  COMPUTER 


Male 

Age  60-69 
Site  epigastric 
Radiation  none 
Duration  7m-lyr 
Pattern  episodic 
Pain  Is  moderate 
Progress  worse 
Aggd  by  food 

COMPUTER  PROBABILITIES 


Relief  antacids 
Nlghtpain  pres. 

Nausea  present 
Vomiting  present 
Meals:  pain  Immed 
Haematemesis  abs 
No  Indigestion 
Bowels  OK 
Micturition  OK 

BASED  ON  THESE  SYMPTOMS 


FUNCTIONAL 
CHOLECYSTITIS 
DUODENAL  ULCER 
GASTRIC  ULCER 
CA.  STOMACH 
none  of  these 

If  you  judge  any  of  the  above  probabilities  to  be  In  error  please 
adjust  them  accordingly. 

PROVISIONAL  DIAGNOSIS  If  appropriate  is— _ 


Level  of  confidence  in  this  diagnosis. 

very  tentative  certain 

1  2  3  4  5 

The  highest  probability  has  been  assigned  to  GASTRIC  ULCER.  If  this  or 
any  other  probability  Is  not  in  accordance  with  your  own  judgement, 
please  Indicate  reasons  for  your  conclusions. 


FIGURE  2-2  Summary  Printout  of  a  Medical  Decision-Aiding  Scheme 
Source:  D.  C.  Barber  and  J.  Fox  (1981). 
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Behavioral  decision  theory  (e.g.,  Elnhorn  and  Hogarth,  1981;  Slovlc, 
et  al.,  1979;  Wallsten,  1980)  has  taken  decision  aiding  out  of  the  realm 
of  mathematics  and  merchandising  into  the  realm  of  behavioral  research  by 
recognizing  the  role  of  judgment  in  structuring  problems  and  In  eliciting 
their  components.  Researchers  In  this  field  have  studied.  In  varying 
degrees  of  detail,  the  psychological  processes  underlying  these  judgments 
and  the  ways  in  which  they  can  be  Improved  through  training,  task 
restructuring,  end  decision-aid  design.  A  particular  focus  has  been  cn 
the  Identification  and  eradication  of  judgmental  biases.  The  research 
described  below  Is  that  which  seems  to  be  needed  to  help  behavioral 
decision  research  fulfill  this  role. 

An  Important  development  In  this  research  over  the  last  decade  has 
been  Its  liberation  from  the  mechanistic  models  of  behavior  Inherited 
from  economics  and  philosophy.  The  result  has  been  more  process-oriented 
theories,  attempting  to  capture  how  people  do  make  and  would  like  to  make 
decisions  (e.g.,  Svenson,  1979).  This  change  was  prompted  In  part  by  the 
realization  that  mechanistic  models  offer  little  Insight  into  central 
questions  of  applications,  such  as  how  action  options  are  generated  and 
when  people  are  satisfied  with  the  quality  of  their  decisions.  These 
developments  are  reflected  in  the  research  described  below. 

There  may  seem  to  be  a  natural  enmity  between  those  purveying 
techniques  of  decision  analysis  and  those  studying  their  behavioral 
underpinnings,  with  the  latter  revealing  the  limits  of  the  procedures 
that  the  former  are  trying  to  sell.  In  general,  however,  there  has  been 
rather  good  cooperation  between  the  two  camps.  Basic  researchers  have 
often  chosen  to  study  the  problems  that  practitioners  find  most 
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troublesome,  and  practitioners  have  often  adopted  basic  researchers'' 
suggestions  for  hov  to  Improve  their  craft.  For  example.  In  both 
commercial  and  government  use,  one  can  find  software  packages  and 
de'clslon-maklng  procedures  that  have  been  redesigned  In  response  to  basic 
research.  Established  channels  (e.g.,  conferences,  paper  distribution 
lists)  exist  for  members  of  this  community  to  communicate  with  one 
another.  Many  of  the  leading  practitioners  have  doctoral-level  training, 
usually  In  psychology,  management  science,  operations  research,  or 
systems  engineering,  and  maintain  academic  contacts.  Indeed,  the 
quantity  of  basic  research  has  been  reduced  by  the  diversion  of  potential 
researchers  to  applied  work,  although  Its  quality  may  have  benefited  from 
being  better  focused.  Although  problems  remain,  research  In  this  area 
has  a  fairly  good  chance  of  being  useful  and  of  being  used.  In  addition, 
none  of  the  research  issues  discussed  In  the  following  sections  appears 
to  pose  any  aerlous  methodological  difficulties.  The  conventional 
experimental  methods  of  the  behavioral  sciences  are  suitable  for 
performing  the  recommended  Investigations. 

RESEARCH  ON  DECISION  MAKING 

Given  the  relatively  good  communication  between  decision-making 
researchers  and  practitioners,  the  primary  focus  of  the  recommendations 
that  follow  is  the  production  of  new  research,  as  opposed  to  its 
dissemination.  It  aeems  reasonable  to  hope  that  the  same  communication 
networks  that  brought  these  applied  problems  to  the  ettention  of 
academics  will  carry  their  partial  solutions  back  to  the  field.  Research 
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on  decision  making  per  se  assumes  that  there  are  general  lesoons  to  be 
learned  from  studying  the  sorts  of  issues  that  recur  in  many  decision 
problems  and  the  responses  typically  made  to  them.  Zn  fact,  the 
complexity  of  real  decision  problems  Is  often  so  great  as  to  prevent  some 
lessons  from  being  learned  from  direct  study. 

These  recommendations  are  cast  In  terms  of  research  needed  to 
Improve  the  use  of  computerized  decision  aids,  referred  to  generically  as 
decision  analysis.  These  aids  work  in  an  interactive  fashion,  asking 
people  to  provide  critical  Inputs  (e.g.,  the  set  of  actions  that  they  are 
considering,  the  probability  of  those  actions  achieving  various  goals), 
combining  those  Inputs  into  a  recommendation  of  what  action  to 
repeating  the  process  until  users  feel  that  they  have  exhausted  its 
possibilities .  In  order  to  be  useful,  an  aid  must:  (a)  deal  with  those 
aspects  of  decision  making  for  which  people  require  assistance,  (b)  ask 
for  inputs  in  a  language  compatible  with  how  people  think  Intuitively 
about  decision  making,  and  (c)  display  its  recommendations  in  a  way  that 
properly  captures  their  lmpllcstlons  and  deflnltlveneas.  Achieving  these 
goals  requires  understanding  of  (a)  how  people  assess  the  quality  of 
human  performance  in  decision-making  tasks,  (b)  the  nature  of 
decision-making  processes,  snd  (c)  how  people  assess  the  quality  of 
declalon-making  processes,  both  those  they  perform  snd  those  performed 
for  them.  The  research  described  below  is  Intended  to  contribute  to  all 
three  of  these  aspects  of  systems  design.  It  is  also  intended  to 
facilitate  the  development  of  supplementary  components  of 
decision-support  systems,  such  as  exercises  for  improving  judgment  or  for 
more  creative  option  generation. 


In  this  light,  research  that  contributes  to  hardware  or  software 
design  should  also  be  a  useful  adjunct  to  any  formal  or  semlformal 
decision-asking  process  in  which  judgment  plays  a  role.  Even  the  devotee 
of  decision  analysis  often  lacks  the  time  or  resources  to  do  anything  but 
an  informal  analysis. 


Decision  Structuring 

Decision  making  is  commonly  characterised  as  involving  the  four 
Interrelated  steps  described  earlier.  The  first  three  of  these  give  the 
problem  its  structure,  by  specifying  the  options,  facts,  and  value  Issues 
to  be  considered  as  well  as  their  interrelations.  Prescriptive  models  of 
decision  making  elaborate  on  the  way  these  steps  should  be  taken.  Most 
descriptive  theories  hypothesize  seme  deviation  of  people's  practice  from 
a  prescriptive  model  (Fischhoff,  Golteln,  and  Shapira,  1931).  These 
deviations  should,  in  principle,  guide  the  development  of  the 
prescriptive  model.  That  is,  they  show  how  the  prescriptive  models  fall 
to  consider  Issues  that  people  want  to  Incorporate  in  their  decisions. 

In  practice,  however,  the  flow  of  information  is  typically  asymmetrical, 
with  prescriptive  models  disproportionately  setting  the  tone  for 
descriptive  research. 

As  a  result,  decision  structuring  Is  probably  the  least  developed 
aspect  of  research  into  both  prescriptive  and  descriptive  aspects  of 
decision  making  (von  Winterfeldt,  1980).  Prescriptive  models  are 
typically  developed  from  the  pronouncements  of  economists  and  others 
regarding  how  people  should  (want  to)  run  their  lives  or  from  ad  hoc 
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lists  of  relevant  considerations.  Descriptive  models  tend  more  or  less 
to  assume  that  these  prescriptions  are  correct.  Neither  seems  to  have 
explored  fully  the  range  of  possible  problem  representations  that  people 
use  when  left  to  their  own  devices. 

Paying  more  attention  to  the  diverse  ways  In  which  people  do  make 
decisions  would  enable  decision  aiders  to  offer  their  clients  a  more 
diverse  set  of  alternative  ways  in  which  they  might  make  decisions,  along 
with  some  elaboration  on  the  typical  strengths  and  weaknesses  of  each 
method.  Some  research  projects  that  might  serve  this  end  follow. 

o  Studies  of  dynamic  structuring,  allowing  for  Iterations  in  the 
decision-making  process,  with  each  round  responding  to  the  insights 
gained  from  Its  predecessors  (Humphreys  and  McFadden,  1980).  Can  people 
use  such  opportunities,  or  do  they  tend  to  stick  to  an  initial 
representation?  Are  there  Initial  structures  that  are  less  confining, 
which  should  be  offered  by  the  aids? 

o  Studies  of  goals  other  than  narrow  optimization.  In  economic 
0 

models,  the  goal  of  decision  making  is  assumed  to  be  maximizing  the 
utility  of  the  Immediate  decision.  Recently  attention  has  turned,  to 
other  goals,  such  as  reducing  the  transaction  costs  from  the  act  of 
making  a  decision,  improving  trust  between  the  individuals  involved  in  a 
decision,  making  do  with  limited  decision-making  expertise,  imposing 
consistency  over  a  set  of  decisions,  or  facilitating  learning  from 
experience.  Theoretical  studies  are  needed  to  clarify  the  consequences 
of  Adopting  these  goals  (e.g.,  how  badly  do  they  sacrifice  optimization); 
empirical  studies  are  needed  to  see  how  often  people  actually  want  to 
accept  them  (particularly  after  they  have  been  informed  of  the  results  of 
the  theoretical  studies). 
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o  Option-generation  studies.  Decision  makers  can  only  choose 
between  the  options  they  can  think  of.  Each  decision  need  not  be  a  new 
test  of  their  Imaginations,  particularly  because  research  indicates  that 
Imagination  often  falls.  Research  can  suggest  better  formulation 
procedures  and  generic  options  that  can  be  built  Into  decision  analysis 
schemes  (Gettys  and  Fisher.  1979). 

o  Many  decision  analysis  schemes  are  sold  as  stand-alone  systems, 
to  be  used  by  decision  makers  without  the  help  of  a  professional  decision 
analyst.  The  validity  of  these  claims  should  be  tested,  particularly 
with  regard  to  decision  structuring,  the  area  in  which  the  largest  errors 
can  occur  (Fits,  et  al.,  1980).  Research  could  also  show  ways  to  Improve 
the  stand-alone  capability  (e.g.,  with  better  introductory  training 
packets). 


Measuring  Preferences 

Unless  one  Is  fortunate  enough  to  find  a  dominating  alternative,  one  that 
Is  better  than  all  competitors  In  all  respects,  making  decisions  means 
making  trade-offs.  When  one  cannot  have  everything,  It  Is  necessary  to 
determine  the  relative  importance  of  different  goals.  Such  balancing 
acts  may  be  particularly  difficult  when  the  question  Is  new  and  the  goals 
that  stand  In  conflict  seem  incommensurable  (Fischhoff,  et  al.,  1980). 
Dealing  with  hasardoua  technologies,  for  example,  leads  us  daily  to  face 
questions  such  as  whether  the  benefits  of  dyeing  one's  hair  are  worth  a 
vague,  minute  Increase  In  the  chances  of  cancer  manv  years  hence. 

Decision  analysis  schemes  seem  to  complicate  life  by  making  these 
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Inherent  conflicts  apparent  (McNeil,  et  al.,  1978).  They  actually 
complicate  It  when  they  pose  these  questions  In  cumbersome,  unfamiliar 
ways  In  order  to  elicit  the  Information  needed  by  their  models — e.g.,  how 
great  an  Increase  In  your  probability  of  being  alive  in  five  years'  time 
would  exactly  compensate  for  the  .20  probability  that  you  will  not 
recover  from  the  proposed  surgery — and  does  this  trade-off  depend  on 
other  factors? 

Such  questions  are  difficult  In  part  because  their  format  Is 
dictated  by  a  formal  theory  or  the  programmer's  convenience,  rather  than 
by  the  decision  maker's  way  of  thinking.  They  are  also  difficult  be^oi";® 
of  the  lack  of  research  guiding  their  formulation.  Research  on  rhc 
elicitation  of  values  has  lagged  behind  research  on  the  elicitation  of 
judgments  of  fact  (Johnson  and  Huber,  1977).  Although  there  are  many 
highly  sophisticated  axiomatic  schemes  for  posing  value  questions,  few 
have  teen  empirically  validated  for  difficult,  real-life  issues.  In 
practice,  perhaps  the  most  common  assumption  Is  that  decision  makers  are 
able  to  articulate  responses  to  any  question  that  is  stated  In  good 
English. 

The  projects  described  below  may  help  solve  problems  that  currently 
are  (or  should  be)  worrying  practitioners.  Some  similar  needs  have  been 
identified  by  the  National  Research  Council's  Panel  on  Survey-Based 
Measures  of  Subjective  Phenomena  (Turner  and  Martin,  198X). 

o  No  opinion.  In  most  behavorial  decision  research,  as  in  most 
survey  research,  economics,  and  preference  theory,  people  are  typically 
assumed  to  know  what  they  want.  Careful  questioning  is  all  that  Is 
needed  to  reveal  the  decision  maker's  Implicit  trade-offs  between 
whatever  goals  are  being  compared.  The  ueed  for  some  response  is  often 
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necessary  for  the  analyala  to  continue*  Knowing  how  to  discover  when 
decision  makers  have  no  opinions  and  how  to  cope  with  that  situation 
would  be  of  great  value.  Studies  of  "no  opinion"  In  survey  research 
(Schumann  and  Presser,  1979)  would  provide  a  useful  base  to  draw  on, 
although  they  often  show  that  people  have  a  disturbing  ability  to 
manufacture  opinions  on  diverse  (and  even  fictitious)  topics. 

o  Interactive  value  measurement.  One  possible  response  to 
situations  In  which  decision  makers*  values  are  poorly  articulated  (or 
nonexistent)  Is  for  the  decision  aider  to  engage  In  a  dialogue  with  the 
client,  suggesting  alternative  ways  of  thinking  about  the  problem  and  the 
implications  of  various  possible  resolutions.  Although  there  are  obvious 
opportunities  for  manipulating  responses  in  such  situations,  research  may 
show  how  they  could  be  minimised;  at  any  rate  they  may  be  rendered  no 
worse  than  the  manipulation  Inherent  in  not  confronting  the  ambiguity  In 
respondents’  values.  Of  particular  Interest  Is  the  question  of  whether 
people  are  more  frank  about  their  values  and  less  susceptible  to  outside 
pressures  when  interacting  with  a  machine  than  with  another  human  being. 
Again,  some  good  leads  could  be  fouud  In  the  survey  research  literature, 
particularly  In  work  dealing  with  the  power  and  prevalence  of  Interviewer 
effect. 

o  Specific  topics.  In  order  to  Interact  constructively  with  their 
clients,  should  decision  alders  be  able  to  offer  a  comprehensive, 
balanced  description  of  the  perspectives  that  one  could  have  on  a 
problem?  The  provision  of  such  perspectives  may  be  enhanced  by  a 
combination  of  theoretical  and  empirical  work  on  how  people  could  and  do 
think  about  particular  issues  (Jungermann,  1980).  For  example,  to  aid 
decision  problems  that  involve  extended  time  hor Isons,  one  would  study 
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how  people  think  About  good  And  bed  outcome*  that  aie  distributed  over 
time.  One  might  discover  that  people  have  difficulty  conceptualising 
distant  consequences  and  therefore  tend  to  discount  them  unduly;  such  a 
tendency  could  be  countered  by  the  use  of  scenarios  that  reify 
hypothetical  future  experiences.  Medical  counseling  and  the  setting  of 
safety  standards  are  two  other  areas  with  specific  problems  that  reduce 
the  usefulness  of  decision  technologies  (e.g.,  the  difficulty  of 
imagining  what:  it  would  be  like  to  be  paralysed  or  on  dialysis, 
unwillingness  to  place  a  value  on  human  life). 

o  Simulating  values.  One  obvious  advantage  of  computerized  *sv« terns 
is  to  work  quickly  through  calculations  using  alternative  valuer, 
different  parameters.  A  possible  didactic  use  would  be  to  help  people 
clarify  what  they  want,  by  simulating  the  implications  of  different  sets 
of  preferences  ("If  those  were  your  trade-offs,  these  would  be  your 
choices"),  both  on  the  problem  in  question  and  on  sample  problems.  Work 
along  this  line  was  done  at  one  time  in  the  context  of  social  judgment 
theory  (Hammond,  1971).  Completing  it  and  making  It  accessible  to  the 
users  of  other  decision  aids  would  be  useful. 

o  Framing.  Recent  research  has  demonstrated  that  formally 
equivalent  waya  of  representing  decision  problems  can  elicit  highly 
inconsistent  preferences  (Kahneman  and  Tversky,  1979;  Tversky  and 
Kahneman,  1931).  Because  most  decision-aiding  schemes  have  a  typical 
manner  of  formulating  preference  questions,  they  may  inadvertently  be 
biasing  the  results  they  produce.  This  work  should  be  continued,  with  an 
eye  to  characterizing  and  studying  the  ways  in  which  decision  analysis 
schemes  habitually  frame  questions. 


Evaluation 


The  decision  maker  looking  for  help  may  be  swamped  by  offers.  The  range 
of  available  options  may  run  from  computerized  decision  analysis  routines 
to  super-soft  decision  therapies.  Few  of  these  schemes  are  supported  by 
empirical  validation  studies;  most  ere  offered  by  Individuals  with  a 
vested  Interest  In  their  acceptance  (Flschhoff,  1980).  A  comprehensive 
evaluation  program  would  help  decision  makers  sort  out  the  contenders  for 
their  attention  and  to  use  those  selected  judiciously,  with  a  full 
understanding  of  their  strengths  and  limitations  (Wardle  and  Wardle, 
1978).  Such  a  program  might  Involve  the  following  elementat 

o  Collecting  and  characterizing  the  set  of  existing  decision  aids 
with  an  eye  to  discerning  common  behavorial  assumptions  (e.g.,  regarding 
the  real  difficulties  people  have  in  making  decisions,  the  ways  in  which 
they  vent  to  have  problems  structured,  or  the  quality  of  the  judgment 
Inputs  they  can  provide  to  decision-making  models). 

o  Examining  the  assumptions  identified  above.  This  might  include 
questions  like:  Can  people  separate  judgments  of  fact  from  judgments  of 
value?  When  decision  makers  are  set  to  act  in  the  name  of  an 
Institution,  can  they  assess  its  preferences,  unencumbered  by  their  own? 
Can  people  Introspect  usefully  about  beliefs  that  have  guided  their  past 
decisions,  free  from  the  biasing  effects  of  hindsight? 

o  Developing  methods  for  evaluating  the  quality  of  decisions  (such 
as  are  produced  by  different  methods).  For  example,  what  weights  should 
be  placed  on  the  quality  of  the  decision  process  and  on  the  quality  of 
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the  outcome  that  arises?  What  level  of  successful  outcomes  should  be 
expected  in  situations  of  varying  difficulty?  This  work  would  he 
primarily  theoretical  (Fischer,  1976). 

o  Clarifying  the  method's  degree  of  determinacy.  To  what  extent  do 
arbitrary  changes  (i.e.,  ones  regarding  which  the  method  is  nilent)  in 
mode  of  application  affect  the  decisions  that  arise  (Hogarth  and 
Makridakls,  1981)?  Similarly,  one  would  like  some  general  guidance  on 
the  sensitivity  of  the  procedure  to  changes  in  various  aspects  of  the 
decision-making  process,  in  order  to  concentrate  efforts  on  the  most 
Important  areas  (e.g«,  problem  structuring  or  value  elicitation). 
Conversely,  one  wants  to  know  how  sensitive  the  method  is  to  the 
particulars  of  each  problem  and  user.  That  is,  does  it  tend  to  render 
the  same  advice  in  all  circumstances? 

o  Assessing  the  impact  of  different  methods  on  "process"  variables, 
such  as  the  decision  maker's  alertness  to  new  information  chat  threatens 
the  validity  of  the  decision  analysis  or  the  degree  of  acceptance  that  a 
procedure  generates  for  the  recommendation  it  produces  (Watson  and  Brown, 
1978).  Such  questioning  of  assumptions  has  been  the  goal  rf  much 
existing  research,  which  should  provide  a  firm  data  base  for  new  work 
(although  many  questions,  such  as  the  first  two  of  the  three  raised,  have 
yet  to  be  studied). 


Improving  Realism 

The  simplified  models  of  the  world  that  decision  analysis  software 
packages  use  to  represent  decision  problems  are  in  at  least  one  key 


respect  very  similar  to  the  models  generated  by  flight  or  weapons 
simulators.  Their  usefulness  Is  constrained  by  the  fidelity  of  their 
representations  to  the  critical  features  of  the  world  they  hope  to 
.model.  Although  there  Is  much  speculation  about  process  effects,  It 
points  in  Inconsistent  directions  and  is  seldom  substantiated  by 
empirical  studies  (either  In  the  laboratory  or  in  operating 
organisations).  Although  these  topics  have  been  studied  very  little  in 
this  context,  research  could  draw  on  whatever  analogous  studies  have  been 
conducted  with  other  kinds  of  simulators.  Some  suggested  research  topics 
follow. 

o  Bot  and  cold  cognition.  Decision  analysis  schemes  are  cold  and 
calculating,  and  they  expect  the  decision  maker  to  be  ao  as  well.  It  Ip 
not  clear  how  well  their  putative  advantages  survive  when  decision  makers 
shift  from  -cold"  to  "hot"  cognition.  Such  s  shift  occurs  with  emotional 
Involvement,  such  as  might  happen  when  the  stakes  increase  or  the  topic 
is  arousing  (Janis  and  Mann,  1977).  The  use  of  decision  aids  for  medical 
patients  pondering  possible  treatments  assumes  that  decision  quality  will 
not  deteriorate  in  such  situations— or  at  least  no  more  than  it 
deteriorates  without  the  aid.  Another  such  shift  involves  time 
pressures,  such  as  might  arise  in  crisis  decision  making  (Wright,  1974). 
Many  proponents  of  decision  analysis  claim  that  time  constraints  actually 
enhance  the  usefulness  of  their  tool,  rather  than  threaten  it,  arguing 
that  a  quick-and-dirty  analysis  is  often  the  most  cost-effective  way  to 
use  the  technology.  Evidence  is  needed  regarding  whether  this  is  true, 
both  when  quickness  is  chosen  and  when  it  is  imposed. 

o  Contingency  planning.  Many  of  the  most  important  uses  of 
decision  aids  are  for  the  sake  of  contingency  planning.  The  essence  of 
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•unh  planning  ia  anticipating  future  situations  and  prescribing  the 
actions  needed  should  they  actually  occur.  In  principle,  preplanning 
responses  should  allow  a  more  leisurely  and  thoughtful  analysis  with 
better  utilisation  of  experts  and  decision  aids  than  would  be  possible  if 
one  waited  until  a  situation  demanding  an  Immediate  response  developed. 
The  success  of  such  efforts  depends  on  the  planner’s  ability  to  imagine 
in  advance  how  various  contingencies  will  appear  should  they  come  about. 
If  the  actual  contingency  does  not  resemble  its  image,  then  the 
(preplanned)  decisions  based  on  that  image  will  seem  inappropriate.  In 
such  cases,  the  decision  maker  must  decide  on  short  notice  whether  to 
adhere  to  the  plan  (and  assume  that  his  or  her  immediate  impress- 
faulty)  or  come  up  with  a  new  plan  on  the  spot  (and  assume  that  the  event 
that  was  anticipated  Is  not  the  event  that  occurred).  Although  the 
stakes  riding  on  contingency  plans  are  often  very  large,  we  have  little 
systematic  knowledge  about  the  correspondence  between  actual  and  planned 
contingencies.  Research  is  needed  on  (1)  when  end  why  situations  look 
(or  feel)  different  when  they  occur  than  they  did  during  planning  and  (2) 
what  to  do  when  plans  made  at  an  earlier  time  seem  inappropriate. 

o  Overriding  recommendations.  The  moment  of  truth  for  the  decision 
aid  comes  when  the  decision  maker  must  decide  to  follow  its 
recommendations  or  override  them.  Analogous  moments  face  the  users  of 
most  other  human-machine  systems,  suggesting  that  the  study  of  overriding 
would  have  broad  implications.  The  research  questions  are:  When  do 
people  even  think  about  overriding?  How  valid  are  the  cues  that  lead 
them  to  do  so?  How  much  better  than  the  aid  are  their  Intuitive 
judgments?  Does  protracted  reliance  on  decision  aids  increase  or 
decrease  intuitive  decision-making  ability?  Existing  research  on  the 
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acceptance  of  computerised  diagnose*  In  medicine *  clinical  psychology * 
and  meteorology  would  provide  a  good  basis  for  this  research* 

o  Better  displays.  Decision  analysts  have  shown  considerable 
.Ingenuity  In  translating  formal  decision  theory  into  terms  that  may  be 
understood  by  less  sophisticated  decision  makers.  More  work  needs  to  be 
done  in  this  area*  particularly  If  decision  aids  are  to  have  stand-alone 
capacity*  The  features  that  the  models  capture  are  a  mixture  of  those 
that  are  easy  to  capture  and  those  that  designers  Intuitively  feel  are 
Important  to  Include.  Each  of  the  four  topics  just  described  In  this 
section  Is  a  factor  that  may  affect  the  realism  of  decision  aids  and*  if 
so*  should  be  considered  in  their  design  and  utilization.  Research 
efforts  to  date  have  hardly  begun  to  tap  the  potential  of  recent  work  in 
computer  graphics  for  developing  superior  displays  (e.g.*  to  facilitate 
interpretation  of  how  robust  a  recommendation  is  by  showing  its 
susceptibility  to  change  with  variation  in  the  values  of  the  input 
parameters).  A  particular  problem  is  that  both  questions  and 
recommendations  typically  appear  without  any  indication  of  their 
rationale.  As  a  result*  decision  makers  may  have  little  feeling  for 
where  the  questioning  is  leading  or  how  robust  the  concluding 
recommendations  will  be  (or  how  they  can  be  explained  to  others). 
Collaborative  efforts  might  increase  both  the  overall  acceptance  of 
decision  analysis  and  the  realism  of  its  recommendations  when  it  is  used. 


Aiding  Diffuse  Decisions 

Common  to  most  decision-making  models  is  the  assumption  that  decisions 
are  made  by  an  identifiable  individual  at  an  identifiable  point  in  time. 
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Clearly,  however,  this  Idealization  often  la  not  realized  in  practice: 
there  may  be  many  parties  to  a  decision;  some  decisions  just  evolve  over 
time  (or  at  least  are  made  to  seem  that  way);  other  decisions  are  made  by 
people  who  do  not  think  of  themselves  as  decision  makers  (e.g., 
supervisors  monitoring  and  directing  the  behavior  of  subordinates  or 
systems);  some  decisions  are  made  by  people  who  are  not  officially 
recognizable  as  decision  makers  (e.g.,  aides  to  a  senior  official)* 

Rather  different  forms  of  research  are  needed  to  improve  decision  making 
in  each  setting;  a  number  of  them  are  outlined  below. 

o  Multiperson  decisions.  Decision  theory  methods  are  typically 
designed  to  explore  and  aggregate  the  beliefs  and  preferences  oi  .  s_  ^ 
individual.  One  approach  to  dealing  with  multiple  decision  makers  is  a 
computational  scheme  for  aggregating  their  beliefs  and  preferences  prior 
to  using  them  in  a  common  decision  model  (Rohrbaugh,  1979).  Theoretical 
work  has  suggested  a  variety  of  analytical  aggregation  schemes.  Although 
this  work  should  continue,  it  could  be  usefully  complemented  by  empirical 
studies  (using  simulations  and  experimentation)  of  how  greatly  the 
results  of  these  various  schemes  differ  and  how  well  they  are  accepted  by 
users.  Another  approach  is  to  have  the  parties  aggregate  their 
perspectives  through  some  structured  interaction  (Sschman,  1975;  Steiner, 
1972).  This  approach,  well  worked  by  students  of  the  risky  shift  and  of 
the  Delphi  methods,  might  benefit  from  research  using  computerized 
systems  that  allow  participants  (perhaps  at  different  sites)  to  go 
through  many  rounds  of  interactions  with  varying  communication  channels 
and  protocols.  For  example,  will  decisions  be  reached  more  quickly  and 
adopted  more  enthusiastically  if  the  parties  can  observe  visual  images  of 
one  another,  not  just  printed  summary  statements? 
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o  Evolving  decision!.  Insofar  as  decisions  represent  choices 
between  alternative  courses  of  action,  any  decision  may  be  expressed  as  a 
statement  of  action  ("I  [or  we]  will  do  X").  Such  translation  of  a 
complex  decision  process  to  Its  procedural  implications  can  have 
drawbacks.  One  Is  that  the  underlying  rationale  of  an  action  is  lost, 
making  It  difficult  to  understand  why  things  are  done  the  way  they  are, 
how  to  respond  to  new  contingencies,  and  when  it  Is  time  to  rethink  the 
whole  decision.  A  second  potential  drawback  Is  that  those  decisions  that 
still  have  to  be  made  are  not  addressed  directly,  leaving  crucial  steps 
to  guesswork  (e.g.,  an  operator  may  be  told  something  to  the  effect  of 
"Figure  out  what  Is  going  on  and  then  follow  steps  to  Sn").  A 
third  possibility  Is  that  procedures  may  have  Internal  Inconsistencies  or 
be  at  cross-purposes,  and  people  either  do  not  realize  It  or  they  realize 
It  but  do  not  quite  know  what  Is  wrong.  Systems  that  add  rules  over  time 
may  be  particularly  prone  to  this  problem  (the  social  security  system  Is 
an  example).  Some  combination  of  artificial  intelligence,  decision 
modeling,  and  experimental  work  might  help  people  to  diagnose  the  logic 
of  the  systems  that  they  deal  with  and  that  they  are  called  on  to 
redesign  (Corbin,  1980;  Klein  and  Weltzenfeld,  1978). 

o  Unwitting  decision  makers.  Just  as  any  decision  may  be  thought 
of  as  an  action,  so  may  each  action  be  thought  of  as  a  decision.  Most 
students  of  decision  making  would  probably  agree  with  the  hypothesis  that 
people  would  be  better  off  If  they  realized  the  decisions  Implicit  in 
their  actions,  and  structured  them  as  such.  For  example,  a  supervisor 
contemplating  the  shutdown  of  a  plant  because  of  a  malfunction  would  make 
wiser  choices  with  even  a  rudimentary  decision  analysis  (l.e.,  listing 
all  possible  courses  of  qctlon,  sketching  out  possible  consequences  and 
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contingencies,  crudely  working  through  the  expected  utility  of  each 
action).  Such  structuring  has  become  part  of  tha  training  of  some 
medical  students.  The  user  of  computerised  Information  retrieval  systems 
(t.g.,  Pres tel,  Teletext)  might  be  usefully  seen  as  making  a  series  of 
decisions  (such  as:  These  alternatives  are  ambiguous — which  gives  me  the 
best  chance  of  getting  the  Information  I  need?  Is  It  worth  my  time  and 
money  to  use  the  system  on  this  problem?  Is  the  answer  I  got  complete 
enough  or  should  I  keep  working?).  A  useful  way  to  exploit  existing 
research  would  be  to  translate  It  into  crude  aids,  adapted  to  the 
conditions  and  problems  of  particular  work  settings  (along  with  an 
evaluation  of  their  efficacy). 

o  Unofficial  decision  makers.  Senior  officials  In  many 
organisations  are  too  busy  to  make  deliberative  analyses  of  the  many 
decisions  they  must  consider.  A  common  (and  sensible)  defense  is  to  have 
aides  conduct  the  analyses.  For  this  stratagem  to  work,  the  senior 
official  must  communicate  well  enough  with  the  aide  to  ensure  that  the 
appropriate  problem  Is  addressed;  the  aide  must  communicate  well  enough 
with  the  senior  official  to  ensure  that  the  rationale  behind  the 
decision-making  method  and  the  Implications  of  Its  conclusions  are 
understood  well  enough  to  be  properly  represented  and  afforded  due 
consideration.  Communication  problems  are  likely  to  be  particularly 
great  when  the  official  must  present  the  conclusions  to  some  larger 
public  or  when  the  training  of  official  and  aide  are  quite  different . 
Consider,  for  example,  the  difficulties  experienced  by  public  officials 
enunciating  the  policies  devised  by  economists  or  by  those  of  junior 
executives  trying  to  sell  decision  analyses  to  old-line  senior 
executives.  Better  methods  of  communication  (and  for  realizing  the  lack 
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Many  formal  and  Informal  processes  in  working  organisations  hinge  on  the 
effective  communication  of  "expert  information.**  Risk  analyses  may 
require  a  metallurgist  to  assess  the  likelihood  of  a  valve's  fracturing 
under  an  anticipated  stress  or  a  human  factora  expert  to  assess  the 
likelihood  of  its  failing  to  open  due  to  faulty  maintenance.  Strategic 
analyses  may  require  substantive  experts  to  assess  the  growth  rate  of  the 
Soviet  economy  or  the  proportion  of  its  expenditures  directed  to  arms. 
Tactical  planning  In  marketing  or  the  military  may  demand  real-time 
reports  by  field  personnel  of  what  seems  to  be  nappenlng  "at  the  front." 
Air  traffic  control  typically  requlrea  succinct,  unambiguous  statue 
reporta  from  all  concerned.  Computerised  career-counseling  routines  or 
procedures  for  establishing  entitlement  to  social  benefits  assume  that 
lay  people  cun  report  on  those  aspects  of  their  own  lives  about  which 
they  are  the  ranking  experts.  The  U.S.  Census  Bureau  makes  similar 
assumptions  when  asking  people  about  their  employment  status,  as  a  step 
toward  directing  federal  policies  and  jebs  programs.  In  product 
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liability  trial*  technical  experts  give  evidence  In  a  highly  stylized 
manner. 

As  can  be  seen  from  these  examples,  experts  may  talk  to  the  consumers 
of  their  advice  directly,  to  ellcitors  who  then  translate  what  they  eay 
into  a  form  usable  by  a  computer,  or  to  a  computer.  Insofar  as  computers 
have  been  designed  by  people,  till  of  these  communication  modes  assume 
(some  fairly  high  level  of  Interpersonal  understanding.  The  ellcitors 
must  ask  questions  that  people  can  sensibly  answer.  The  recipients  of 
those  answers  must  Interpret  them  with  an  appreciation  of  the  errors  and 
ambiguities  they  may  conceal.  The  quality  of  that  communication  is 
likely  to  depend  on  the  novelty  of  the  problems,  the  historic  level  or 
Interaction  between  questioner  and  answerer,  and  the  quickness  with  which 
miscommunicatlons  produce  diagnostic  signs.  Poor  elicitation  by  air 
traffic  controllers  may  become  visible  very  quickly;  whereas  employment 
surveys  may  (and  have)  elicited  biased  responses  and  misdirected  economic 
planning  for  years  without  the  error's  being  detected.  Particularly 
clumsy  elicitation  may  lead  users  to  reject  the  eliciting  system,  thereby 
avoiding  mistakes  but  also  wasting  the  resources  that  have  been  Invested 
in  its  design. 

New  research  about  elicitation  and  the  translation  of  existing 
research  findings  into  more  usable  form  could  benefit  a  wide  variety  of 
enterprises.  As  this  chapter  discusses,  elicitation  is  not  e  field  of  ' 
inquiry  or  application  in  and  of  Itself,  but  a  function  that  recurs  in 
many  problems.  This  creates  special  difficulties  for  the  accumulation 
and  dissemination  of  knowledge  about  it. 
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BACKGROUND 


Perhaps  because  elicitation  is  a  part  of  many  problems  but  all  of  none. 

It  has  emerged  neither  as  a  discipline  nor  as  an  area  that  is  seen  to 
require  special  expertise*  The  typical  assumption  is  that  elicitation  is 
not  a  particular  problem,  as  long  as  things  stay  fairly  simple  and  one 
uses  common  sense*  The  validity  of  that  assumption  may  not  be  questioned 
until  some  egregious  problem  has  clearly  arisen  from  a  particular 
failure.  When  problems  arise,  the  lack  of  a  coherent  body  of  knowledge 
may  encourage  ad  hoc  solutions,  with  little  systematic  testing  or 
accumulation  of  knowledge.  Solutions  are  generated  from  the  resources  of 
those  working  on  a  particular  problem  and  viewed  from  their  narrow 
perspective. 

One  reason  for  aggregating  these  elicitation  issues  into  e  single 
chapter  is  to  keep  them  from  being  orphaned,  as  parts  of  many  problems 
for  which  there  is  no  focus  of  responsibility.  Another  reason  is  to 
suggest  that  there  are  enough  recurrent  themes  to  generate  s  coherent 
body  of  knowledge,  thereby  reducing  the  degree  to  which  each  system 
designer  faced  with  an  elicitation  problem  must  start  from  scratch. 
Although  work  may  still  focus  on  specific  problems,  conceptualising  them 
In  a  general  way  may  Increase  both  the  pool  of  talent  they  draw  on  and 
tha  breadth  of  perspective  with  which  their  solutions  are  interpreted  and 
reported.  Because  a  common  element  of  these  projects  is  dealing  with 
substantive  experts,  their  cumulative  Impact  should  be  to  generate  a 
better  understanding  of  the  judgmental  processes  of  experts. 
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The  research  bases  for  Che  following  projects  are  sufficiently 
diverse  that  further  details  are  given  within  each  context.  Zn  some 
cases,  there  Is  a  distinct  research  literature  on  which  new  projects  can 
be  based.  In  others,  the  proposed  topic  does  not  exist  as  a  separate 
pursuit,  or  at  least  not  within  the  context  of  human  factors;  tha 
literature  cited  Is  suggestive  of  the  kinds  of  approaches  that  have 
proven  useful  In  other  fields  or  related  problems  that  might  be  drawn  on. 


RESEARCH  ON  ELICITATION 


Ensuring  a  Common  Frame  of  Reference 

An  obvious  precondition  for  communication  Is  ensuring  that  ellcltor  and 
respondent  are  talking  about  the  same  thing.  In  ordinary  conversation 
the  participants  have  some  opportunity  for  detecting  and  rectifying 
misunderstandings.  If  questions  are  set  down  once  for  all  respondents, 
then  misunderstandings  must  be  anticipated  in  advance*  Some  implicit 
theory  of  potential  (misinterpretations  must  guide  the  question 
composers  for  management  systems,  accident  report  forms,  or  automatic 
diagnostic  routines  that  rely  on  expert  judgment. 

These  problems  are  not,  of  course,  unique  to  human  factors.  They  are 
probably  best  understood  by  professionals  whose  central  concern  for  the 
longest  periods  of  time  hca  been  asking  questions;  these  Include 
anthropologists  (Agar,  1980),  linguists,  historians  (Hexter,  1971), 
survey  researchers  (Payne,  1952) ;  philosophers,  and  some  social 
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psychologists  (Rosenthal  and  Rosnow,  1969).  Two  general  conclusions  that 
ons  can  derive  from  their  work  la  that  the  opportunities  for 
misinterpretation  are  ouch  greater  than  most  people  would  presuppose  and 
that  the  nature  of  possible  specific  nlslnterpretatlons  is  hard  to 
Imagine  Intuitively. 

The  chances  for  olscoaounlcatlon  are  likely  to  Increase  to  the  extent 
that  ellcltor  and  respondent  come  from  different  cultures  and  have  had 
little  opportunity  to  Interact.  Systens  designed  by  technical  experts 
for  lay  users  often  fall  into  this  category,  especially  when  the 
elicitation  Is  far  removed  physically  or  temporally  from  the  design 
effort*  Consider,  for  example,  a  computerised  job  search  program  that 
requires  unemployed  workers  to  characterize  their  experience  In  terms  of 
one  of  the  12,000  categories  of  the  Dictionary  of  Occupational  Titles 
(DOT)  code  (e.g.,  handkerchief  presser).  Although  a  considerable 
Intellectual  effort  has  gone  Into  imposing  a  semblance  of  order  on  the 
world  of  work,  that  order  nay  be  very  poorly  matched  to  the  way  In  which 
applicants  conceptualize  their  experience.  Indeed,  even  those  who  elicit 
such  Information  from  job  applicants  and  translate  It  Into  the  DOT  code 
on  a  full-time  basis  may  have  considerable  difficulty.  Similar  problems 
may  face  a  system  designed  to  clarify  entitlement  to  social  services  or  a 
computerized  system  for  diagnosing  car  or  radio  problems  on  the  basis  of 
a  user's  description  of  presenting  symptoms.  These  problems  may  persist 
even  with  the  dearest  display  and  the  most  lucid  users'  manual. 

Although  the  details  of  each  problem  are  unique,  seeing  their  common 
elements  can  enable  designers  to  exploit  a  larger  body  of  existing 
research  and  research  methods.  One  strategy  is  literature  reviews  that 
make  accessible  the  methods  used  by  fields  such  as  anthropology  to 
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uncover  misunderstandings.  Using  these  oethods  with  smell  samples  of 
users  prior  to  designing  systems  or  in  the  early  stages  of  design  could 
effectively  suggest  minor  changes  or  even  major  Issues  (such  as  whether 
the  system  could  ever  stand  alone,  or  whether  it  will  always  need  an 
Interpreter  between  it  and  the  actual  user)*  Such  strategies  are 
Increasingly  being  used  in  survey  design;  they  may  even  lead  to  some 
revision  In  the  categories  of  Justice  Department  statistics  so  as  to  make 
them  more  compatible  with  the  ways  in  which  victims  of  crimes  think  about 
their  experience  (National  Research  Council,  1976)* 

Another  research  strategy  is  to  review  existing  case  studies  of 
mishaps  (e.g.,  in  diplomacy,  survey  research,  police  work,  or  software 
design)  for  evidence  of  problems  due  to  questioners  and  respondents 
unwittingly  speaking  different  languages  (Brooks  and  Ballar,  1978).  Such 
studies  would  help  establish  the  prevalence  of  such  problems  and  create  a 
stock  of  cautionary  tales  for  educational  and  motivational  purposes. 

A  third  strategy  involves  experimental  and  observational  studies  of 
groups  of  individuals  who  regularly  communicate  with  one  another,  in 
order  to  see  how  well  they  understand  one  another's  perspectives. 

Software  designers  and  less  educated  users,  engineers  and  machine 
operators,  and  market  researchers  and  consumers  are  a  few  such  dyads. 

The  Intuitive  beliefs  of  the  ellcltors  in  each  of  these  dyads  regarding 
the  perspectives  of  their  respondents  might  provide  some  productive 
hypotheses  and  reveal  some  misconceptions  worthy  of  correction. 

Better  ways  of  eliciting  information  should  also  suggest  better  ways 
of  presenting  it.  Informing  and  counseling  patients  about  medical  risks 
is  one  area  in  which  these  problems  are  currently  under  active  study  (see 
Chapter  2). 
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Matching  Questions  to  Mental  Structural 

A  presumption  of  many  allcltatlon  efforts  is  that  the  respondent  has  an 
answer  to  any  question  that  the  ellcitor  can  raise  (Turner  and  Martin, 
198X).  One  contributing  factor  to  this  belief  la  the  fact  that  ellcltors 
often  cannot  accept  "no  answer"  for  an  answer,  needing  some  best  guess  at 
the  answer  in  order  to  get  on  with  business.  A  second  contributing 
factor  may  be  the  tendency,  long  known  to  surveyors,  for  respondents  to 
offer  opinions  on  even  nonexistent  Issues,  perhaps  reflecting  some 
feeling  that  they  can,  should,  or  must  have  opinions  on  everything.  A 
third  factor  may  be  the  ellcltors*  (intuitive  or  scientific)  models  of 
memory  that  presume  a  coherent  store  of  knowledge  waiting  to  be  tapped  by 
whatever  question  proves  most  useful  to  the  ellcitor  (Llndley,  et  al., 
1979). 

Coping  with  situations  in  which  the  respondent  has  little  or  no 
knowledge  about  the  topic  in  question  is  dealt  with  in  the  next  section, 
on  how  to  elicit  assessments  of  Information  quality.  Alternatively,  the 
respondent  may  have  the  needed  information,  but  not  in  the  form  required 
by  the  question.  Whenever  there  is  Incompatibility  between  the  way  in 
which  knowledge  is  organized  and  the  way  in  which  it  is  elicited,  the 
danger  arises  that  the  expert  may  not  be  vised  to  best  advantage,  may 
provide  misleading  information,  or  may  be  seduced  into  doing  a  task  to 
which  his  or  her  expertise  does  not  extend.  For  example,  risk  assessment 
programs  often  require  the  designers  of  a  technics]  system  to  describe  it 
in  terms  of  the  logical  interrelationships  between  various  components 
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(Including  it#  human  operator#,  repair  people,  supplier#,  etc.)  and  to 
asses#  the  probability  of  these  components'  falling  at  various  rates, 
perhaps  as  a  function  of  several  variables  (Jennergren  and  Keeney, 

1981).  Given  these  Judgmental  Inputs,  these  programs  may  perform 
miraculous  simulations  and  calculations;  however,  the  value  of  such 
analyses  is  contingent  on  the  quality  of  the  judgments.  The  processes  by 
which  experts  are  recruited  may  or  may  not  take  into  consideration  the 
need  for  these  special  skills.  In  some  situations,  no  one  may  have 

Research  designed  to  improve  the  compatibility  of  questions  with  the 
way  in  which  knowledge  is  stored  should  be  guided  by  substantive  th*c a 
about  that  storage  as  well  as  practical  knowledge  of  the  Information 
needed.  The  citations  given  here  represent  different  approaches  to 
conceptualizing  such  mismatches  between  precise  questions  and  differently 
organized  or  unorganized  knowledge.  As  an  example  of  the  kinds  of 
testable  hypotheses  that  emerge  from  these  literatures,  consider  the 
possibility  that  many  experts  experience  the  topics  of  their  expertise 
one  by  one,  whereas  ellcitors  often  need  a  summary  (e.g.,  of  the  rate  of 
target  detections  by  sonar  operators,  the  conditional  probability  of 
misreading  an  altimeter  given  a  particular  number  of  hours  of  flying 
experience,  the  distribution  of  hearing  deficits  associated  with  various 
noise  levels).  If  experts  are  not  accustomed  to  aggregating  their 
experience,  then  they  will  respond  differently  to  procedures  that  request 
aggregate  estimates  immediately  and  those  that  focus  first  (and  perhaps 
entirely)  on  the  recall  of  individual  incidents  (Fischhoff  end  Whipple, 
1981).  This  particular  research  could  build  somewhat  on  probability 
learning  studies  or  attempts  to  distinguish  between  episodic  and  semantic 
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Effort*  to  design  the  beet  response  node  assume  that  respondents  have 
the  knowledge  that  the  ellcltor  needs,  but  not  organised  in  the  aost 
convenient  form.  A  more  troublesome  situation  arises  when  they  do  not 
have  it  organised  at  all.  Zn  that  case  the  elicltor's  task  becomes  to 
evoke  all  of  the  relevant  bits  and  pieces,  then  devise  seme  scheme  for 
Interpreting  them.  Doing  so  first  requires  discovering  that  Incoherence 
exists,  which  may  not  be  easy,  Insofar  as  a  set  of  questions  may  elicit 
consistent  responses  simply  because  it  has  consistently  imposed  one  of 
several  possible  perspectives.  Although  sensitive  elicltors  may  already 
be  poking  around  creatively,  there  are  few  codified  and  tested 
procedures.  Such  procedures  might  Involve  standard  sets  of  questions 
designed  to  produce  diverse  perspectives,  which  the  respondent  would  then 
Integrate  to  provide  a  best  guess  (or  set  of  best  guesses)  for  the 
problem  at  hand.  For  example,  one  might  always  ask  about  case-by-case 
and  aggregate  estimates,  in  that  order.  Such  efforts  might  also  prompt 
and  be  helped  by  the  development  of  memory  models  allowing  for  multiple. 
Incoherent  representations. 


Clarifying  Information  Quality 

Before  taking  action  on  an  expert's  opinion,  one  wants  to  know  how  good 
that  best  guess  is.  Great  uncertainty  might  prompt  one  to  try  to  uncover 
its  sources  or  to  take  alternative  courses  of  action  (e.g.,  hedging  one's 
bets).  Although  explicit  assessments  of  uncertainty  are  becoming  a 
greater  part  of  enterprises  such  ss  risk  analysis  (Frirley,  1977), 
weather  forecasting  (Murphy  and  Winkler,  1977),  and  strategic  assessment 
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(Daly  and  Andriole ,  1980),  such  experiences  are  rare  for  most  people.  As 
one  would  expect  In  novel  elicitation  situations,  the  responses  that 
people  give  sre  not  always  to  be  trusted.  Assessments  of  information 
quality  (or  confidence  or  probability)  have  been  the  subject  of  extensive 
research  over  the  last  decade  (Lichtenstein,  et  al.,  1982).  It  has 
produced  a  fairly  robust  set  of  methods  for  eliciting  uncertainty  and  a 
moderately  good  understanding  of  human  performance  in  this  regard.  The 
clearest  finding  is  that  people  have  a  partial  but  not  complete 
appreciation  of  the  extent  of  their  own  knowledge.  Most  commonly,  this 
partial  knowledge  expresses  itself  in  overconfidence,  which  seems  qu;..r« 
impervious  to  most  attempts  at  debiasing,  except  for  intensive  training 
(Fischhoff,  1982). 

Many  practical  problems  could  be  solved  in  this  area  with  a  moderate 
investment  in  completing  the  research  that  has  already  been  started. 

This  research  could  use  the  stock  of  elicitation  techniques  already 
available  to  understand  better  the  range  and  potency  of  overconfidence 
biases,  to  clarify  how  worrisome  they  are,  and  to  determine  the  most 
effective  training  and  how  far  it  can  be  generalized.  Of  particular 
interest  is  the  extent  to  which  experts  are  prone  to  these  problems  when 
making  judgments  in  their  areas  of  expertise;  current  evidence  suggests 
that  they  are,  but  it  is  still  inconclusive  given  the  importance  of  the 
question  (Spetzler  and  Stael  von  Holstein,  1975). 

The  practical  steps  that  can  be  taken  subsequent  to  such  research  are 
developing  and  testing  training  procedures,  identifying  the  least 
bias-prone  elicitation  methods  for  situations  in  which  training  is 
impossible  or  ineffective,  and  anticipating  the  extent  of  bias  with 
different  methods  and  situations  in  order  to  apply  ad  hoc  corrections. 
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Choosing  between  these  steps  and  Implementing  them  efficiently  will 
require  a  more  detailed  understanding  of  the  cognitive  processes  Involved 
In  representing  and  Integrating  probabilistic  Information.  Although 
existing  research  covers  much  of  the  ground  between  basic  cognitive 
psychology  and  field  applications,  it  has  not  quite  touched  bases  with 
either  extreme.  Coping  with  this  practical  problem  might  provoke  some 
interesting  theoretical  work  In  the  representation  of  knowledge. 


Eliciting  Systems 

In  the  examples  used  in  the  preceding  sections,  the  knowledge  that 
experts  were  asked  to  provide  dealt  with  the  components  of  some  large 
system  (e.g.,  a  failure  probability,  a  job  choice,  a  burnout  rate).  At 
times,  however,  experts  are  required  to  describe  the  entire  system 
(Hayes-Roth,  et  al.,  1981).  Software  packages  that  attempt  to  elicit  a 
big  picture  Include  some  of  those  used  In  decision  structuring,  failure 
probability  modeling  (U.S.  Nuclear  Regulatory  Commission,  1981),  map 
making,  route  planning,  and  economic  analysis.  Once  such  systems  have 
been  programmed  well  enough  to  work  at  all,  one  must  ascertain  the  degree 
of  fidelity  between  the  representations  they  produce  and  the  conceptual 
or  physical  systems  they  are  meant  to  model;  attempts  to  develop  better 
elicitation  methods  or  to  cope  with  known  limits  or  errors  should  follow 
(Brown  and  Van  Lehn,  I960).  The  research  strategies  outlined  below, 
based  In  part  on  the  initial  work  already  begun  and  in  part  on 
discussions  with  troubled  system  ellcltors,  may  shed  some  light  on  these 
problems.  In  each  case  pne  would  want  to  know  whether  a  change  In 


procedure  made  a  difference  and,  if  so,  whether  one  method  would  be 
preferred  In  some  or  all  situations.  Because  so  little  systematic 
knowledge  la  available  on  how  results  may  vary  with  different  elicitation 
procedures,  generalizing  the  existing  research  findings  should  be  done 
cautiously. 

o  Determining  whether  formally  equivalent  ways  of  eliciting  the 
same  information  produce  different  responses.  For  example,  a  category  of 
events  may  be  judged  differently  when  considered  as  a  whole  and  when 
disaggregated  into  component  categories. 

o  Evaluating  the  effectiveness  of  methods  that  require  more  an., 
less  "deep”  (or  analytical  or  inferential)  judgments  about  system 
operation.  For  example,  if  a  process  produces  a  distribution  of  events 
(e.g.,  failure  rates),  one  could  assess  that  distribution  directly  or 
judge  something  about  the  data-generating  process. 

o  Varying  the  amount  of  feedback  provided  about  how  the  elicited 
system  operates.  For  example,  when  a  simulation  of  an  Industrial  process 
Is  designed  according  to  an  expert's  judgment.  It  may  be  run  a  few  times, 
just  to  see  if  It  produces  more  or  less  sensible  results.  The  expert 
could  then  Introduce  apparently  needed  adjustments.  Such  tinkering 
should  lead  to  successive  Improvements  in  the  model;  however,  it  can  also 
prevent  simulations  from  producing  nonlntuitlve  (i.e.,  surprising, 
Informative)  results.  It  also  threatens  the  putative  independence  of  the 
models  created  by  different  experts  in  areas  such  as  climatology  and 
macroeconomics.  The  convergence  of  these  models'  predictions  (about  the 
future  of  the  economy,  for  example)  is  used  as  a  sign  of  their  validity. 
In  practice;  however,  econometricians  monitor  o^e  another's  models  and 
adjust  theirs  if  they  produce  outlying  predictions. 
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o  Assessing  experts'  ability  to  judge  the  completeness  of  s 
representation.  How  well  can  they  tell  whether  all  Important  components 
have  been  Included?  Available  evidence  suggests  that  considerations  that 
are  out  of  sight  are  also  out  of  mind;  once  experts  have  begun  tw  think 
about  a  model  In  a  particular  way,  the  accessibility  of  other 
perspectives  Is  apprecially  reduced  (Fischhoff,  et  al.,  1978).  If  this 
is  generally  true,  an  elicltor  might  try  to  evoke  a  variety  of 
perspectives  on  the  system  superficially  before  pursuing  any  in  depth  (as 
a  sort  of  intra-expert  brainstorming). 


Estimating  Numerical  Quantities 

A  common  form  of  uncertainty  is  knowing  something  about  a  topic,  but  not 
a  necessary  fact.  If  that  fact  la  a  number  (e.g.,  the  number  of  tanks  an 
enemy  has  or  the  percentage  of  those  tanks  that  are  in  operating  order), 
it  may  be  possible  to  use  the  related  fscts  in  a  systematic  way  if  one 
can  devise  a  rule  or  algorithm  for  composing  them  (Armstrong,  1 977).  The 
validity  of  such  estimates  depends  on  the  appropriateness  of  the 
algorithms,  the  quality  of  the  component  estimates,  and  the  accuracy  of 
their  composition.  Used  appropriately,  algorithms  can  make  otherwise 
Impenetrable  judgmental  processes  explicit  and  subject  both  to  external 
criticism  and  to  self-improvement,  as  one  can  systematically  update  one's 
best  estimate  whenever  more  is  learned  about  any  component  (Singer,  1971). 

Although  there  are  many  advocates  of  algorithmic  thinking  and 
anecdotal  evidence  of  its  power,  there  do  cot  seem  to  be  many  empirical 
studies  uf  their  usefulness  (Hogarth  and  Makrldakis,  1981).  Such 
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of  algorithm  efficacy  aa  do  exist  seem  concentrated  on  the  solving  of 
deterministic  logical  problems  for  which  all  relevant  evidence  la 
presented  to  the  respondent  and  a  clear  criterion  of  success  exists, 
rather  than  estimation  tasks  in  which  the  accuracy  of  the  estimate  will 
be  unclear  until  some  external  validation  is  provided.  Like  any  other 
judgmental  technique,  algorithmic  thinking  could  be  more  trouble  than  it 
is  worth  if  it  increases  confidence  in  judgment  more  than  it  Improves 
judgment. 

A  primary  research  project  here  would  be  to  compile  a  set  of 
plausible  and  generally  applicable  algorithmic  strategies.  Process, 
tracing  of  the  judgmental  processes  of  expert  estimators  might  be  one 
source.  The  algorithms  discovered  in  the  study  of  logical  problem 
solving  might  be  another.  A  subsequent  project  could  attempt  to  teach 
people  to  us?  these  algorithms,  then,  looking  at  the  fidelity  with  which 
they  can  be  applied,  measure  the  accuracy  of  their  results  and  their 
influence  on  confidence.  The  uae  of  multiple  algorithms  and  people's 
ability  to  correct  the  results  of  Imperfect  algorithms  are  also  worth 
study.  The  best  algorithms  could  then  become  part  of  management 
information  systems,  decision  support  systems,  and  the  like. 

Two  interpretive  literature  reviews  might  provide  useful  adjuncts  to 
this  research.  One  woulu  look  at  work  on  mental  arithmetic  of  the  sort 
required  when  people  must  execute  algorithms  in  their  heads.  Although 
computational  devices  should  be  able  to  eliminate  the  need  for  such 
exercises,  judges  may  still  be  caught  without  their  tools  or  may  use 
unwritten  mini-algorithms  in  order  to  produce  component  estimates  (once 
they've  gotten  the  general  idea).  The  second  review  would  summarize,  in 
a  form  accessible  to  designers,  the  psychophysics  literature  on 
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stlfflulus~presentatlon  and  response-mode  affecta  (Poultou,  1977).  That 
literature  shows  the  degree  of  variability  In  magnitude  estimation  that 
can  arise  from  "artlfactual"  changes  In  procedure  (e.g.,  order  of 
alternative  presentation,  kind  of  numbers  used). 


Detecting  Reporting  Bias 

The  preceding  sections  have  assumed  that  elicitor  and  respondent  are 
engaged  In  an  honest,  unconflicted  attempt  to  produce  a  best  estimate  of 
some  quantity  or  relationship.  When  research  Identifies  difficulties, 
one  assumes  a  mutual  good  faith  effort  on  the  part  of  ellcitors  and 
experts  to  eliminate  them.  In  the  real  world,  however,  many  wrong 
answers  are  deliberate;  their  producers  do  not  wish  to  have  them  either 
detected  or  corrected.  If  the  citations  given  here  are  at  all 
representative,  systematic  misrepresentation  has  been  of  greatest 
interest  to  those  concerned  with  the  social  and  economic  context  within 
which  behavior  takes  place.  Such  misrepresentations  may  be  usefully 
divided  Into  two  categories.  The  first  includes  deliberate  attempts  to 
deceive  In  order  to  gain  some  advantage.  For  example,  economists 
chronically  mistrust  verbal  reports  o*  people’s  preferences  (i.e., 
surveys)  for  fear  that  respondents  engage  in  strategic  behavior,  trying 
to  "put  one  over"  on  the  questioner  and  distort  the  survey's  results 
(Brookshire,  et  al.,  1976).  Some  critics  of  survey  research  are  even 
advocating  that  respondenta  do  ao  deliberately  ao  as  to  stop  the  survey 
juggernaut  (see  Turner  and  Martin,  198X),  as  do  some  people  in 
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organisations  who  feel  threatened  by  computerised  Information  systems  end 
wish  to  eee  them  fail* 

The  second  category  of  alsreporta  reflects  cultural  or  subcultural 
norms.  Xn  a  business  or  military  unit,  for  example «  optimism  (or 
grousing)  may  be  the  norm  for  communication  between  members  of  some  ranks 
(Tlhansky,  1976).  Or  there  may  be  a  norm  of  exaggerating  one's  wealth  or 
weight.  Those  who  share  the  norms  know  how  to  recode  the  spoken  word  to 
gain  a  more  accurate  assessment;  however ,  mechanical  systems  designed  by 
people  outside  the  culture  may  take  those  reports  at  face  value  and 
thereby  Introduce  systematic  errors  into  their  workings. 

Although  investigating  misreporting  is  likely  to  be  quite  difficult., 
identifying  it  is  part  of  systems  design*  One  way  to  start  is  to  review 
the  relevant  literature  in  fields  that  have  dealt  with  these  questions 
(e.g.,  sociology,  economics).  A  second  is  to  interview  experts  off  the 
record  about  how  (and  how  often)  they  try  to  manipulate  systems  that  pose 
questions  to  them.  A  third  is  to  observe  ongoing  elicitations  for  which 
it  is  possiblle  to  validate  responses. 

Difficulties,  once  identified,  must  still  be  treated.  One  method  is 
to  institute  penalties  for  misreporting.  A  second  is  to  make  consistency 
checks  to  detect  errors.  A  third  is  to  eliminate  the  reasons  for 
misreporting  (e.g.,  ensuring  confidentiality).  A  fourth  is  to  correct 
mlareports  for  known  biases.  For  example,  the  Central  Electricity 
Generating  Board  in  Great  Britain  discovered  that  it  could  quite 
accurately  predict  the  time  needed  to  return  a  power  station  to  operation 
by  doubling  the  time  estimates  reported  by  the  chief  plant  engineers. 

One  difficulty  with  such  adjustments  is  that  people  may  change  their 
reporting  practices  if  they  find  out  about  them  (Kidd,  1970). 
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Reporting  Rest  Events 

Many  planning  and  design  activities  are  heavily  gulderl  by  reports  of  past 
events,  particularly  accidents  cr  other  failures  (Petzcldt,  1977; 
Rasmussen,  1980).  One  reconstructs  the  way  in  which  a  system  should  have 
operated,  contrasts  that  with  the  way  in  which  it  actually  operated,  and 
uses  that  comparison  to  Improve  future  design  (perhaps  assigning  guilt 
and  enacting  penalties  along  the  way). 

Such  retrospections  are  inevitably  colored  by  the  reporter's 
knowledge  ox  what  has  happened.  As  common  sense  suggests  and  the 
citations  below  partially  document,  that  coloring  can  be  the  source  of 
needed  detail  or  of  systematic  distortion.  Zt  has  been  found,  for 
example,  that  people  seem  to  exaggerate  in  hindsight  what  could  have  been 
(and  was)  known  in  foresight;  they  use  explanatory  schemes  so  complicated 
and  so  poorly  specified  as  to  defy  empirical  test;  they  remember  people 
as  having  been  more  like  their  present  selves  than  was  actually  the  case; 
they  fall  to  remember  crucial  acts  that  they  themselves  performed.  These 
problems  seem  to  afflict  both  the  garden-variety  retrospections  evoked  in 
laboratory  studies  and  those  of  professional  historians,  strategic 
analysts,  and  eyewitnesses  (Fischhoff,  1975). 

One  needed  project  is  to  make  these  studies  available  to  those 
engaged  in  eliciting  or  using  retrospective  reports.  Another  is  to 
attempt  to  replicate  them  in  human  factors  domains.  Of  particular 
interest  are  esses  in  which  the  direction  of  bias  has  been  documented 
sufficiently  to  allow  recallbratlon  of  biased  retrospections.  In  cases 
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in  which  distortions  sre  less  predictable,  techniques  should  be  developed 
to  help  experts  reconstruct  their  view  of  the  situation  before,  during, 
and  after  the  event.  For  example,  such  research  may  show  that  people 
exaggerate  the  probability  they  assigned  (or  would  have  assigned)  to  past 
events  before  they  occurred  by  about  20  percent,  on  the  average.  That 
knowledge  may  make  it  possible  to  adjust  retrospective  probability 
assessments,  but  not  to  eliminate  distortions  in  the  way  particular 
events  and  causal  links  are  drawn. 

For  assigning  blame  or  understanding  how  an  accident  situation  looked 
to  an  operator  just  before  things  started  to  gc  wrong,  strict  (accuiate) 
reconstruction  is  essential.  For  understanding  how  the  system  actually 
operates,  one  needs  to  be  wary  of  the  danger  that  experts  have  learned 
too  much  from  a  particular  event,  thereby  misinterpreting  the  Importance 
and  generality  of  the  causal  forces  involved.  Generals  who  prepare  for 
the  last  war  may  fit  this  stereotype,  as  may  the  operators  of  supervisory 
control  systems  who  respond  to  each  mishap  by  ensuring  that  it  will  not 
happen  again,  then  rest  confident  that  the  system  as  a  whole  is  now 
fail-safe. 

Three  research  strategies  appear  to  offer  some  promise  for  clarifying 
these  questions.  One  is  to  review  the  reports  of  historians,  judges, 
journalists,  and  others  about  how  they  detect  and  avoid  biases.  A  second 
is  to  do  theory-based  experiments,  looking  at  how  memory  accommodates  new 
information,  particularly  to  see  which  processes  are  reversible.  The 
third  is  research  on  debiasing,  looking  at  the  effect  of  directly  warning 
people,  of  raising  the  stakes  riding  on  a  decision,  or  of  instructing 
them  to  change  the  structure  of  the  task  to  ooe  that  uses  their 
Intellectual  skills  to  better  advantage. 


CONCLUSION 


Eliciting  information  from  experts  successfully  is  important  to  a  variety 
of  systems  and  organisations.  The  care  taken  in  elicitation  varies 
greatly,  from  detailed  studies  of  the  elicitation  of  some  specific 
recurrent  judgments,  to  careful  deliberations  unsupported  by  empirical 
research,  to  casual  solutions.  Even  though  elicitation  is  not  a 
discipline  per  se,  research  such  as  that  suggested  in  this  chapter  could 
focus  more  attention  on  it  and  make  a  body  of  knowledge  accessible  to 
designers.  In  part,  that  knowledge  would  be  borrowed  from  related  fields 
(with  suitable  translations);  in  part  it  would  be  created  expressly  to 
solve  human  factors  problems.  Some  of  these  projects  could  be  undertaken 
in  their  own  right;  others  would  be  best  developed  as  part  of  ongoing 
projects,  with  more  emphasis  on  elicitation  than  might  otherwise  be  the 
case.  The  interdisciplinary  aspect  of  many  projects  may  generate 
Interest  in  human  factors  problems  on  the  part  of  workers  in  other  fields 
(e.g.,  memory  representation,  workplace  culture),  and  their  expertise 
could  contribute  to  human  factors  research. 
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SUPERVISORY  CONTROL  SYSTEMS 


In  the  past  15  years  the  introduction  of  automation  into  working 
environments  has  created  more  and  more  jobs  in  which  operators  are  given 
very  high  levels  of  responsibility  and  very  little  to  do.  The  degree  of 
responsibility  and  the  amount  of  work  vary  from  position  to  position,  but 
the  defining  properties  of  such  jobs  are:  (1)  The  operator  has  overall 
responsibility  for  control  of  a  system  that,  under  normal  operating 
conditions,  requires  only  occasional  fine  tuning  of  system  parameters  in 
order  to  maintain  satisfactory  performance.  (2)  The  major  tasks  are  to 
program  changes  in  Inputs  or  control  routines  and  to  serve  as  a  backup  in 
the  case  of  a  failure  or  malfunction  in  a  system  component.  (3) 

Important  participation  In  system  operation  occurs  Infrequently  and  at 
unpredictable  times.  (4)  The  time  constraints  associated  with 
participation,  when  it  occurs,  can  be  very  short,  of  the  order  of  a  fe;< 
seconds  or  minutes.  (5)  The  values  and  costs  associated  with  operator 
decisions  can  be  very  large.  (6)  Good  performance  requires  rapid 
assimilation  of  large  quantities  of  Information  and  the  exercise  of 
relatively  complex  Inference  processes. 


The  principal  authors  of  this  chapter  are  Thomas  B.  Sheridan,  Baruch 
Flschhoff,  Michael  Posner,  and  Richard  V.  Pew. 


These  kinds  of  jobs  are  found  In  the  process  control  Industries,  such 
as  chemical  plants  and  nuclear  power  plants*  They  are  Involved  In  the 
control  of  aircraft,  ships,  and  urban  rapid  transit  systems,  robotic 
remote  control  systems  for  inspection  and  manipulation  In  the  deep  ocean, 
and  computer-aided  manufacturing.  They  are  involved  In  mecical 
patient-monitoring  systems  and  lav  enforcement  Informat*  >>.,  and  coac  .- ,1 
systems.  As  computer  aids  are  introduced  Into  military  commanc' 
control  systems,  such  jobs  become  involved  in  that  area.  For  example, 
the  Army  alone  currently  has  70  automated  or  computer-aided  systems  ..t 
the  concept  development  stage  (U.  S.  Army  Research  Institute,  1179;  The 
other  services  have  similar  projects  under  development. 

The  human  factors  problems  involved  in  supervisory  control  systems 
can  be  classlfed  Into  five  categories. 

1.  Display.  In  the  past  these  systems  have  used  large  arrays  of 
meters  and  gauges  or  large  situation  boards  and  control  panels  to  display 
Information,  with  the  general  £oal  of  displaying  everything,  because  one 
never  knows  exactly  what  will  be  needed.  Little  attention  has  been  paid 
to  the  need  to  assimilate  diverse  information  sources  into  coherent 
patterns  for  making  inferences  simply  and  directly.  Today  computers  are 
being  used  more  and  more  in  the  control  of  these  operations;  large 
display  panels  are  being  collapsed  into  computer-generated  displays  that 
can  call  up  the  needed  information  on  demand.  These  developments  in 
physical  technology  are  pushing  human  factors  engineers  to  devise  better 
ways  of  coding  and  formating  large  collections  of  information  to 
facilitate  interpretation  ana  reliable  decisions  &y  operators.  Also 
aeeded  are  better  means  of  accessing  information,  means  that  are  not 


3 


opaque  and  do  not  leave  operators  confused  in  urgent  end  stressful 
situations. 

2.  Command.  The  emergence  of  powerful  computers  and  robotic  devices 
h'«s  necessitated  the  development  of  better  "command  languages,"  by  which 
operators  can  convey  instructions  to  a  lower-level  Intelligence,  perhaps 
giving  examples  or  hints  and  providing  criteria  or  preferences,  and  doing 
it  in  a  communication  mode  that  is  natural  and  adaptable  to  different 
people  and  linguistic  styles. 

3.  Operator's  Model.  We  also  lack  well-developed  methodologies  for 

identifying  the  Internal  conceptual  model  on  the  basis  of  which  an 
operator  attempta  to  solve  a  problem.  (This  has  also  been  called  the 
opt  /stem  -i.<iage,  picture,  or  problem  space.)  Incorrect 

operator's  models  can  lead  to  disastrous  results  (e.g..  Three  Mile 
Island);  it  is  obviously  o  matter  of  utmost  importance  fur  oper*».or*>  of 
military  command  and  control  systems  to  acquire  proper  conceptual  models 
and  keep  them  updated  on  a  moment-by-moment  basis  in  times  of  crisis. 

4.  Workload.  We  have  no  good  principles  of  Job  design  for 
operations  in  supervisory  control  systems,  in  part  because  it  has  proved 
extremely  difficult  to  measure  or  estimate  the  mental  workloads 
involved.  They  tend  to  be  highly  transient,  varying  from  light  and 
boring  when  the  work  is  routine  to  extremely  demanding  when  action  is 
critical.  At  present  there  is  no  consensus  on  what  mental  workload  is  or 
how  to  measure  it,  especially  in  the  context  of  supervisory  control. 

5.  Proficiency  and  Error.  Issues  of  training  and  proficiency 
maintenance  are  critical  in  this  kind  of  operation  because  each  event  is 
in  some  sense  unique  and  is  drawn  from  an  extremely  large  set  of 
possibilities,  most  of  which  will  never  occur  during  the  operating  life 
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of  the  system.  It  is  Dot  easy  to  anticipate  what  types  of  errors  will 
occur  or  how  to  train  to  prevent  them. 

SUPERVISORY  CONTROL  IN  DIFFERENT  APPLICATIONS 

This  section,  adapted  from  Sheridan  (1982),  provides  brief  comparisons 
and  contrasts  among  different  applications  of  supervisory  control 
systems:  process  control,  vehicle  control,  and  manipulators. 


Process  Control 

The  term  process  usually  refers  to  a  dynamic  system,  such  as  a  fossil 
fue?  or  nuclear  power  generating  plant  or  a  chemical  or  oil  production 
facility,  that  Is  fixed  in  space  and  operates  more  or  less  continuously 
in  time.  Typically  time  constants  are  slow— many  minutes  or  hours  may 
elapse  after  a  control  action  is  taken  before  most  of  tKv  -.yetem  response 
is  complete. 

Most  such  processes  involve  large  structures  with  fluids  flowing  from 
one  place  to  another  and  involve  the  use  of  heat  energy  to  effect  the 
fluid  or  vice  versa.  Typically  such  systems  involve  multiple  personnel 
and  multiple  machines,  and  at  least  some  of  the  people  move  from  one 
location  of  the  process  to  another.  Usually  there  is  a  central  control 
room  where  many  measured  signals  are  displayed  and  where  valves,  pumps, 
and  other  devices  are  controlled. 
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Supervisory  control  has  been  emerging  as  an  element  In  process 
control  for  several  decades.  Starting  with  electromechanical  controllers 
or  control  stations  that  could  be  adjusted  by  the  operator  to  maintain 
certain  variables  within  limits  (a  home  thermostat  Is  an  example), 
special  electronic  circuits  gradually  replaced  the  electromechanical 
function.  In  such  systems  the  operator  can  become  part  of  the  control 
loop  by  switching  to  manual  control.  Usually  each  control  station 
displays  both  the  variable  being  controlled  (e.g.,  room  temperature  for 
the  thermostat)  and  the  control  signal  (e.g.,  the  flow  of  heat  from  the 
furnace).  Many  such  manual  control  devices  may  be  lined  up  in  the 
control  room,  together  with  manual  switches  and  valves,  status  lights, 
dials  and  recording  displays,  and  as  many  as  1,500  alarms  or 
annunciators— windows  that  light  up  to  indicate  what  plant  variable  has 
just  gone  above  or  below  limits.  From  the  pattern  of  these  alarms  (e.g., 
500  In  the  first  minute  of  a  loss-of-coolant  accident  and  800  in  the 
second  minute,  by  recent  count,  in  a  large  new  nuclear  plant)  the 
operator  la  supposed  to  divine  what  is  happening. 

The  large,  general-purpoce  computer  has  found  its  way  into  process 
control.  Instead  of  multiple,  ludependent,  conventional 
proportional-integral-derivative  controllers  for  each  variable,  the 
computer  can  treat  the  set  of  variables  as  a  vector  and  compute  the 
control  trsjectory  that  would  be  optimal  (in  the  sense  of  quickest,  most 
sfflcient,  or  whatever  criterion  is  important).  Because  there  are  many 
mors  interactions  than  the  number  of  variables,  the  variety  of  displayed 
signals  and  the  number  of  possible  adjustments  or  programs  the  human 
operator  may  input  to  the  computer-controller  are  potentially  much 
greater  than  before.  Thys  there  la  now  a  great  need,  accelerated  since 
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the  events  at  Three  Mile  Island,  to  develop  displays  that  integrate 
complex  patterns  of  information  and  allow  the  operator  to  issue  commands 
in  a  natural,  efficient,  and  reliable  manner.  The  term  system  state 
vector  is  a  fashionable  way  to  describe  the  display  of  minimal  chunks  of 
information  (using  G.  A.  Miller's  well-known  terminology)  to  convey  more 
meaning  about  the  current  state  vector  of  variables,  where  it  has  been  in 
the  past,  and  where  it  is  likely  to  go  in  the  near  future. 


Vehicle  Control 

Unlike  the  processes  described  above,  vehicles  move  through  space  and 
carry  their  operators  with  them  or  are  controlled  remotely.  Various 
types  of  vehicles  have  come  under  a  significant  degree  of  supervisory 
control  in  the  last  30  years. 

We  might  start  with  spacecraft  because,  in  a  sense,  their  function  Is 
the  simplest.  They  are  launched  to  perform  well-defined  missions,  and 
their  interaction  with  their  environment  (other  than  gravity)  is  nil.  In 
other  words,  there  are  no  obstacles  and  no  unpredictable  traffic  to  worry 
about.  It  was  in  spacecraft,  especially  Apollo,  that  human  operators  who 
were  highly  skilled  at  continuous  manual  control  (test  pilots  or  ” Joy 
stick  jockeys'*)  had  to  adapt  to  a  completely  new  way  of  getting 
information  from  the  vehicle  and  giving  it  commands — this  new  way  was  to 
program  the  computer.  The  astronauts  bad  to  learn  to  use  a  simple 
keyboard  with  programs  (different  functions  appropriate  to  different 
mission  phases),  nouns  (operands,  or  data  to  be  addressed  or  processed) 
and  verbs  (operations,  or  actions  to  be  performed  on  the  nouns). 


Of  course,  the  astronauts  still  performed  a  certain  number  of 
continuous  control  functions*  They  controlled  the  orientation  of  the 
vehicle  and  maneuvered  It  to  accomplish  star  sighting,  thrust, 
rendezvous,  and  lunar  landing.  But,  as  is  not  generally  appreciated  by 
the  public,  control  In  each  of  these  modes  was  heavily  aided*  Not  only 
were  the  manual  control  loops  themselves  stabilized  by  electronics,  but 
also  nonmanual,  automatic  control  functions  were  being  simultaneously 
executed  and  coordinated  with  what  the  astronauts  did* 

In  commercial  and  military  aircraft  there  has  been  more  and  more 
supervisory  control  In  the  last  decade  or  two*  Commercial  pilots  are 
called  flight  managers,  indicative  of  the  fact  that  they  must  allocate 
their  attention  among  a  large  number  of  separate  but  complex 
computer-based  systems.  Military  aircraft  are  called  flying  computers, 
and  Indeed  the  cost  of  the  electronics  in  them  now  far  exceeds  the  cost 
of  the  basic  airframe*  By  means  of  inertial  measurement,  a  feature  of 
the  new  jumbo  jets  as  well  as  of  military  aircraft,  the  computer  can  take 
a  vehicle  to  any  latitude,  longitude,  and  altitude  within  a  fraction  of  a 
kilometer*  In  addition  there  are  many  other  supervisory  command  modes 
Intermediate  between  such  high-level  commands  and  the  lowest  level  of 
pure  continuous  control  of  ailerons,  elevators,  and  thrust.  A  pilot  can 
set  the  autopilot  to  provide  a  display  of  a  smooth  command  course  at 
fixed  turn  or  climb  rates  to  follow  manually  or  can  have  the  vehicle 
slaved  to  this  course.  The  eutopilot  can  be  set  to  achieve  a  new 
altitude  on  a  new  heading*  The  pilot  can  lock  onto  radio  beams  or  radar 
signals  for  automatic  landing.  In  the  Lockheed  L-1011,  for  example, 
there  are  at  least  10  separate  Identifiable  levels  of  control.  It  Is 
important  for  the  pilot  to  have  reliable  means  of  breaking  out  of  tiic.ec. 
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automatic  control  mode*  and  ravartlng  to  manual  control  or  aoaa 
Intermediate  mode.  For  example,  when  In  an  automatic  landing  mode  the 
pilot  can  either  push  a  yellow  button  on  the  control  yoke  or  Jerk  the 
yoke  back  to  manually  get  the  aircraft  back  under  direct  control. 

Air  traffic  control  poses  Interesting  supervisory  control  problems, 
for  the  headways  (spacing)  betweeu  aircraft  in  the  vicinity  of  major 
commercial  airports  are  getting  tighter  and  tighter,  and  efforts  both  to 
save  fuel  and  to  avoid  noise  over  densely  populated  urban  areas  require 
more  radical  takeoff  and  landing  trajectories.  New  computer-based 
communication  aids  will  supplement  purely  verbal  communication  br-<-'  -•»; 
pilots  and  ground  controllers,  and  new  display  technology  will  help  the 
already  overloaded  ground  controllers  monitor  what  is  happening  in 
three-dimensional  space  over  larger  areas,  providing  predictions  of 
collision  and  related  vital  information.  The  CDTI  (cockpit  display  of 
traffic  Information)  is  a  new  computer-based  picture  of  weather,  terrain 
hazards  such  as  mountains  and  tall  structures,  course  Information  such  as 
way  points,  radio  beacons  and  markers,  and  runways  and  command  flight 
patterns  as  well  as  the  position,  altitude,  heading  (and  even  predicted 
position)  of  other  aircraft.  It  makes  the  pilot  less  dependent  on  ground 
control,  especially  when  out-the-wlndow  visibility  is  poor. 

More  recently  ships  and  submarines  have  been  converting  to 
supervisory  control.  Direct  manual  control  by  experienced  helmsmen, 
which  sufficed  for  many  years,  has  been  replaced  both  by  the  installation 
of  Inertial  navigation,  which  calls  for  computer  control  and  provides 
capability  never  before  available,  and  by  the  trends  toward  higher  speed 
and  long  time  lags  produced  by  larger  size  (e.g.,  the  new  supertankers). 
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New  autopilots  and  computer-based  display  aids,  similar  to  those  In 
aircraft,  are  now  being  used  In  ships. 

Manipulators  and  Discrete  Parts  Handling 

In  a  sense,  manipulators  combine  the  functions  of  process  control  and 
vehicle  control.  The  manipulator  base  may  be  carried  on  a  spacecraft,  a 
ground  vehicle,  or  a  submarine,  or  its  base  may  be  fixed.  The  hand 
(gripper,  end  effector)  Is  moved  relative  to  the  base  in  up  to  three 
degrees  of  translation  and  three  degrees  of  rotation.  It  may  have  one 
degree  of  freedom  for  gripping,,  but  some  hands  have  differentially 
movable  fingers  or  otherwise  hav?  more  degrees  of  freedom  to  perform 
special  cutting,  drilling,  finishing,  cleaning,  welding,  paint  spraying, 
sensing,  or  other  functions. 

Manipulators  are  being  used  In  many  different  applications.  Including 
lunar  moving  vehicles,  undersea  operations,  and  hazardous  operations  In 
Industry.  The  type  of  supervisory  control  and  its  justification  differs 
according  to  the  application. 

The  fact  of  a  three-second  time  delay  In  the  earth-lunar  control  loop 
resulting  from  round-trip  radio  transmission  from  earth  leads  to 
instabilities,  unless  an  operator  waits  three  seconds  after  each  of  a 
series  of  Incremental  movements.  This  makes  direct  manual  control 
time-consuming  and  Impractical.  Sheridan  and  Ferrell  (1967)  proposed 
having  a  computer  on  the  moon  receive  commands  to  complete  segments  of  a 
movement  task  locally  using  local  sensors  and  local  romputer  program 
control.  They  proposed  calling  this  mode  supervisory  control.  Delays  In 
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control  loops  that  report  position  and  velocity.  If  the  parts  conveyor 
la  sufficiently  reliable,  welding  or  painting  nonexistent  objects  seldom 
occurs,  so  that  more  sophisticated  feedback,  involving  touch  or  vision, 

Is  usually  not  required.  Manufacturing  assembly,  however,  has  proven  to 
be  a  far  more  difficult  task. 

In  contrast  to  assembly  line  operations,  in  which,  even  If  there  Is  a 
mix  of  products,  every  task  Is  prespecified.  In  many  new  applications  of 
manipulators  with  supervisory  control,  each  new  task  Is  unpredictable  to 
considerable  extent.  Some  examples  are  mining,  earth  moving,  building 
construction,  building  and  street  cleaning  and  maintenance,  trash 
collection,  logging,  and  crop  harvesting,  in  which  large  forces  and  power 
must  be  applied  to  external  objects.  The  human  operator  is  necessary  to 
program  or  otherwise  guide  the  manipulator  In  some  degrees  of  freedom,  to 
accomodate  each  new  situation;  in  other  respects  certain  characteristic 
motions  are  preprogrammed  and  need  only  to  be  Initiated  at  the  correct 
time.  In  some  medical  applications,  such  as  microsurgery,  the  goal  is  to 
minify  rather  than  enlarge  motions  and  forces,  to  extend  the  surgeon's 
hand  tools  through  tiny  body  cavities  to  cut,  to  obtain  tissue  samples, 
to  remove  unhealthy  tissue,  or  to  stitch*  Again,  the  surgeon  controls 
some  degrees  of  freedom  (e.g.,  of  an  optical  probe  or  a  cauterizing 
snare),  while  automation  controls  other  variables  (e.g.,  air  or  water 
pressure)* 
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THEORY  AND  METHOD 

There  are  a  number  of  limited  theories  and  methods  In  the  human  factors 
literature  that  should  be  brought  to  bear  on  the  use  of  supervisory 
control  systems.  A  great  deal  remains  to  be  done,  however,  to  apply  them 
In  this  context.  The  discussion  that  follows  deals  with  five  aspects  of 
the  problem.  The  first  considers  current  formal  models  of  supervisory 
control.  The  second  discusses  display  and  command  problems.  The  third 
takes  up  computer  knowledge-based  systems  and  their  relation  to  the 
Internal  cognitive  model  of  the  operator  for  on-line  decision  making  In 
supervisory  control.  The  fourth  deals  with  mental  workload,  stress,  and 
research  on  attention  and  resouxce  allocation  as  they  relate  to 
supervisory  control.  The  fifth  Is  concerned  with  Issues  of  human  error, 
system  reliability,  trust,  and  ultimate  authority. 


Modeling  Supervisory  Control 

In  the  area  of  real-time  monitoring  and  control  of  continuous  dynamic 
processes,  the  optimal  control  model  (Baron  and  Kleinman,  1969)  describes 
the  perceptual  motor  behavior  of  closed-loop  systems  having  relatively 
short  time  constants.  Experimentation  on  this  topic  has  been  limited, 
suggesting  that  this  class  of  model  may  bs  broadened  to  represent 
monitoring  and  discrete  decision  behavior  in  dynamic  systems  In  which 
control  Is  Infrequent  (Levison  and  Tanner,  1971).  There  are  also 
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attempts  to  extend  this  work  to  explore  Its  applicability  to  more  complex 
systems  (Baron,  et  el.,  1981;  Kok  and  Stassen,  1980). 

An  increasing  number  of  supervisory  control  systems  can  be 
represented  by  a  hierarchy  of  three  kinds  of  interaction  (Sheridan, 

1982):  (1)  a  human  operator  interacting  with  a  high-level  computer,  (2) 

low-level  computers  interacting  with  physical  entities  in  the 
environment,  and  (3)  the  resulting  multilevel  and  multiloop  Interaction, 
having  interesting  symmetrical  properties  (Figure  4-1).  Since  there  are 
three  levels  of  intelligence  (one  human,  two  artificial),  the  allocation 
of  cognitive  and  computational  tasks  among  the  three  becomes  central. 
Using  Rasmussen's  (1979)  categorization  of  behavior  Into  knowledge-based, 
rule-based,  and  skill-baaed  behavior,  the  operator  may  assign  rule-based 
tasks  (e.g.,  pattern  recognition,  running  planning  and  predictive  models, 
organizing)  to  the  high-level  computer  (Figure  4-2).  Similarly, 
•kill-based  tasks  (filtering,  display  generation,  servo-control)  may  be 
assigned  to  various  low-level  computers.  The  operator  must  concentrate 
on  the  environmental  tasks  that  compete  for  his  attention,  allocating  his 
attention  among  five  roles:  (1)  planning  what  to  do  next,  (2)  teaching 
or  on-line  programming  of  the  computer(s),  (3)  monitoring  the  (semi) 
automatic  behavior  of  the  system  for  abnormalities,  (4)  intervening  when 
necessary  to  make  adjustments,  maintaining,  repairing,  or  assuming  direct 
control,  and  (5)  learning  from  experience. 


SEMI-AUTOMATIC  SUBSYSTEM  (SAS)  HUMAN  INTERACTIVE  SUBSYSTEM  (HIS) 
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1.  Tatk  It  observed  directly  by  humtn  operator. 

2.  Talk  ta  observed  indirectly  through  tan  ion, 
computer*,  and  displays.  Thi*  SAS  feedback 
interact*  with  that  from  within  HIS. 

3.  Task  it  controlled  within  SAS  automatic  mode. 

4.  Tatk  it  affected  by  the  procat*  of  being 
tamed. 

5.  Tatk  effects  actuator*  and  in  turr,  it  affected. 

6.  Human  operator  directly  affect*  tatk. 

7.  Human  operator  affeett  tatk  indirectly  tnroutfh 
control*,  HIS  computer*,  and  actuators.  This 
control  interacts  with  that  from  within  SAS. 

8.  Human  operator  gett  feedback  I,  vithin 
HIS. 

9.  Human  operator  adjusts  cor... v  j.* — t, 

10.  Human  operator  adjusts  display  ptram.ner«. 


FIGURE  4-1  Multiloop  Interaction  in  a  Supervisory  Control  System 
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Human  Operator 


Multilevel  Allocation  of  Tasks  in  a  Supervisory  Control  System 


FIGURE  4-2 
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Display  and  Command 

Design  of  integrated  computer-generated  displays  is  not  a  new  problem, 
and  the  military  services  and  space  agencies  have  pioneered  developments 
in  this  area  for  aircraft  and  various  command  and  control  systems.  But 
the  technology  continues  to  create  more  possibilities.  Operators  of 
supervisory  control  systems  need  to  have  fewer  displays,  not  more, 
telling  them  what  they  want  or  need  to  know  when  they  want  or  need  lo 
know  it.  An  additional  design  problem  is  that  what  operators  think  tr— 
need  and  what  they  really  need  may  differ. 

As  computer  collaborators  become  more  and  more  sophisticated  a  useful 
type  of  display  would  tell  the  operator  what  the  computer  knows  and 
assumes,  both  about  the  system  and  about  the  operator,  and  what  it 
Intends  to  do. 

An  Important  source  of  guidance  regarding  the  design  of  displays  has 
been  and  will  continue  to  be  the  intuitive  beliefs  of  experienced 
operators.  The  designer  needs  to  know  how  much  credence  to  give  to  these 
intuitions.  Too  little  attention  may  mean  forfeiting  a  valuable  source 
of  information;  too  much  may  result  in  inappropriate  designs  that  fit 
untested  folk  wisdom  (a  pilot's  belief  in  the  value  of  verisimilitude -in 
displays  is  an  example  of  the  latter  problem).  Ericsson  and  Simon's 
taxonomy  (1980)  of  situations  in  which  introspection  is  more  and  less 
valid  is  one  point  of  departure  for  research.  Studies  of  metacognition, 
people's  understanding  cf  their  own  cognitive  processes  (as  contrasted 
with  current  psychological  understanding),  are  a  second  (Cavanaugh  and 


Borkowski,  1980).  The  studies  of  clinical  judgment  conducted  In  the 
1950s  and  1960s  (Goldberg,  1968)  are  a  third.  These  studies  found  that 
in  the  course  of  their  diagnoses  expert  clinicians  imagine  that  they  rely 
on  more  variables  and  use  them  In  more  complex  manner  than  appears  to  be 
the  case  from  attempts  to  model  their  diagnostic  processes. 

Although  good-quality  computer-generated  speech  is  both  available  and 
cheap,  and  although  It  can  give  operators  warnings  and  other  information 
without  their  prior  attention  being  directed  to  it,  little  Imaginative 
use  of  such  a  capability  has  been  made  as  yet  In  supervisory  control. 

The  use  of  command  language  has  arisen  more  recently  In  conjunction 
with  teaching  or  programming  robot  systems.  A  more  primitive  form  of  It 
Is  found  In  the  new  autopilot  command  systems  In  aircraft.  Giving 
commands  to  a  control  system  by  means  of  strings  of  symbols  In  syntax  Is 
a  new  game  for  most  operators.  Progress  In  this  area  depends  on  careful 
technology  transfer  from  data  processing  that  is  self-paced  to  dynamic 
control  In  which  the  pace  is  determined  by  many  factors.  Naturalness  In 
use  of  such  language  Is  also  an  Important  goal. 

Command,  In  many  circumstances.  Is  not  a  solitary  task.  The  operator 
must  Interact  with  many  Individuals  In  order  to  get  a  job  done.  This  may 
be  particularly  the  case  when  the  nature  of  the  emergency  means  that  the 
technical  system  cannot  be  trusted  to  report  and  respond  reliably — that 
Is,  an  Interacting  human  system  may  assume  (and  perhaps  interface  with) 
some  of  the  functions  of  the  Interacting  technical  system.  The  kinds  of 
human  Interaction  possible  Include  requesting  information,  monitoring  the 
response  of  the  system,  notifying  outsiders  (e.g.,  for  evacuation,  to 
provide  special  skills),  and  terminating  unnecessary  communications. 

When  sre  these  Interactions  initiated?  How  valid  are  the  cues?  Wliat 


feature*  of  technical  systems  ask*  such  Intervention  more  and  lesa 
feasible?  How  does  having  others  around  affect  operators'  thoughts  and 
actions  (e.g.,  are  they  more  creative,  more  risk-averse,  more  careful)? 

Another  question  that  arises  with  multi person  systems  is  whether  one 
Individual  (or  group)  should  both  monitor  for  and  cope  with  crises*  In 
medicine  It  la  not  always  assumed  that  the  same  individual  has  expertise 
In  both  diagnosis  and  treatment*  perhaps  In  supervisory  control  systems 
the  equivalent  functions  should  be  separated,  and  different  training  anu 
temperament  called  for  In  monitoring  and  In  Intervention. 

Computer  Knowledge-Based  Systems 
and  the  Operator's  Internal  Cognitive  Model 

It  Is  not  a  new  Idea  that,  in  performing  a  taak,  people  somehow  represent 
the  task  In  their  heads  and  calculate  whether,  given  certain  constraints, 
doing  this  will  result  in  that.  Such  Ideas  derive  from  antiquity. 

Human-Machine  Control 

In  the  1950s  the  development  of  the  "observer"  In  control  systems  theory 
formalized  this  Idea.  That  Is,  a  differential  equation  model  of  the 
external  controlled  process  is  Included  In  the  automatic  controller  and 
is  driven  by  the  same  Input  that  drives  the  actual  process.  Any 
discrepancy  between  the  output  of  this  computerized  model  of  the 
environmental  process  and  the  actual  process  is  fed  back  as  a  correction 


to  th«  internal  model  to  force  its  variables  to  be  continuously  the  same 
as  the  actual  process.  Then  any  and  all  state  variables  as  represented 
(observed)  in  the  Internal  model  may  be  used  to  directly  control  the 
process,  if  direct  measurement  of  those  same  variables  in  the  actual 
environment  may  be  costly,  difficult,  or  impossible.  This  physical 
realization  of  the  traditional  idea  of  the  internal  model  probably 
provoked  much  of  the  current  research  in  cognitive  science. 

Running  in  fast-time,  updating  initial  conditions  at  each  of  a 
succession  of  such  calculations,  the  model  becomes  a  "predictor  display" 
that  provides  the  operator  with  a  projection  of  what  will  happen  under 
given  assumptions  of  Input  (Kelly,  1968).  Further  comparisons  can  be 
made  between  outputs  of  such  real-time  models  run  in  the  computer  and 
those  of  the  operator's  own  Internal  model,  not  only  for  control  but  also 
for  failure  detection  and  Isolation  (Sheridan,  1981).  Teach  has 
developed  a  realization  of  this  as  an  operator  aid  for  application  to 
process  control  (Tsach  et  al.„  1982). 

Ideally  the  computer  should  keep  the  operator  Informed  of  what  it  Is 
assuming  and  computing,  and  the  operator  should  keep  the  computer 
Informed  of  what  he  or  she  is  thinking. 

Cognitive  Science 

In  the  last  several  years  cognitive  psychology  has  contributed  some 
theories  about  human  inference  that  make  the  application  of 
knowledge-based  systems  psrtlcularly  relevant  to  supervisory  control. 

The  idea  is  that  reasoning  and  decision  making  consist  of  the  developing 
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and  searching  of  complex  problem  spaces  (Newell  and  Simont  1972)  and  of 
applying  one  or  more  inference  procedures  about  information  in  a 
knowledge  base  that  represents  the  decision  maker's  understanding  of  the 
situation  (Collins  and  Loftus,  1975)*  This  is  similar  to  but  more 
inclusive  and  less  well  developed  than  the  Internal  process  model  used  by 
control  theorists.  Rasmussen's  (1979)  qualitative  model  of  human 
decision  making  about  process  control  is  entirely  compatible  with  this 
view.  And,  the  contribution  of  specialises  in  artificial  lnteH^^n..  ' 
concerning  knowledge-based  systems  provides  one  way  to  Implement  th^ 
computer  portion  of  such  human-computer  interaction. 

A  number  of  human  factors  problems  relate  to  people's  ability  tr  be... 
in  mind  the  basic  workings  of  a  complex  system  and  to  update  that  view 
depending  on  the  current  state  of  the  system.  Recent  studies  of 
cognitive  processes  in  skilled  operators  such  as  taxi  drivers  (Chi  et 
al.,  1980)  or  chess  players  (Chase  and  Simon,  1973)  begin  to  provide  the 
kind  of  information  that  will  be  needed  by  human  factors  designers 
evaluating  these  issues.  For  example,  how  can  people  best  be  trained  to 
develop  effective  problem  spaces?  What  is  the  optimal  mix  of  analog  and 
digital  representation?  How  can  the  computer's  data  base  system  be  used 
to  aid  the  individual  in  developing  and  updating  of  such  an  internal 
model?  What  means  can  be  used  to  ensure  that  the  current  state  of  the 
model  fits  with  the  current  state  of  the  system?  With  what  frequency 
should  a  person  be  interrogated  about  his  or  her  current  view  of  the 
model  to  make  sure  that  he  or  she  is  still  "with  it"  in  control  of  the 
system?  For  human  supervision  to  be  really  effective,  a  detailed 
understanding  of  how  the  human  controller  grasps  a  complex  system  at  any 
moment  in  time  and  updates  it  over  time  is  necessary. 
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How  can  ve  determine  a  given  operator's  Internal  cognitive  model  of  a 
given  task  at  a  given  time?  One  method  is  to  ask  the  operator  to  express 
It  In  natural  language,  but  the  obvious  difficulty  is  that  each 
-operator's  expression  is  unique,  making  it  very  difficult  to  measure 
either  discrepancy  from  reality  or  to  compare  across  operators.  Verbal 
protocol  techniques  (Bainbrldge,  1974)  make  use  of  key  words  and 
relations.  More  formal  psychometric  techniques  (multiattribute  utility 
assessment,  conjoint  or  multidimensional  scaling,  Interpretive  structural 
modeling,  policy  capturing,  and  fuzzy  set  theory)  offer  some  promising 
ways  of  telling  a  computer  one's  knowledge  and  values  in  structural  form. 

A  likely  (and  perhaps  common)  source  of  difficulty  is  a  mismatch  in 
the  mental  models  of  a  system  of  those  who  design  it  and  those  who 
operate  it.  Operators  who  fail  to  recognize  this  disparity  are  subject 
to  unpleasant  surprises  when  the  system  behaves  in  unexpected  ways. 
Operators  who  do  recognize  it  may  fail  to  exploit  the  full  potential  of 
the  system  for  fear  of  surprises  if  they  push  it  into  unfamiliar 
territory  (Young,  1981).  On  a  descriptive  level,  it  would  be  useful  to 
understand  the  correspondence  between  the  mental  models  of  designers  and 
operators  as  well  as  to  know  which  experiences  signal  operators  that 
there  is  a  mismatch  and  how  they  cope  with  that  information.  On  a 
practical  level,  it  would  be  useful  to  know  more  about  the  possibility  of 
improving  the  match  of  these  two  models  by  steps  such  as  involving 
operators  more  in  the  design  process  or  showing  them  how  the  design 
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evolved  (rather  than  giving  them  a  reconstruction  of  its  final  state). 

The  magnitude  of  these  problems  is  likely  to  grow  to  the  extent  that 
designers  and  operators  have  different  training,  experience,  and 
Intensity  of  involvement  with  systems. 
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Mental  Workload 

The  concept  of  mental  workload  aa  discussed  In  this  section  Is  not  unique 
to  supervisory  control,  but  it  is  sufficiently  Important  In  this  context 
to  be  included  here  as  a  special  consideration. 

Human-Machine  Control  (This  section  Is  adapted  from  Sheridan  and  Youn? , 
1982). 

During  the  last  decade  "mental  workload"  has  become  a  concept  of  great 
controversy,  not  because  of  disagreement  over  whether  it  is  Important, 
but  because  of  disagreement  over  how  to  define  and  measure  It.  Military 
specifications  for  mental  workload  are  nevertheless  being  prepared  by  the 
Air  Force,  based  on  the  assumption  that  mental  workload  measures  will 
predict — either  at  the  design  stage  or  during  a  flight  or  other 
operation — whether  an  operation  can  succeed.  In  other  words,  it  is 
believed  that  measurements  of  mental  workload  are  more  sensitive  in 
anticipating  when  pilot  or  operator  performance  will  break  down  than  are 
conventional  performance  measures  of  the  human-machine  system. 

At  the  present  time  "mental  workload"  is  a  construct.  It  must  be 
inferred;  it  cannot  be  observed  directly  like  human  control  response  or 
system  performance,  although  it  might  be  defined  operationally  in  terms 
of  one  or  several  or  a  battery  of  tests.  There  is  a  clear  distinction 
between  mental  and  physical  workload:  The  latter  is  the  rate  of  doing 
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mechanical  work  and  expending  calories.  There  la  consensus  on 
measurements  based  on  respiratory  gases  and  other  techniques  for 
aieasurlng  physical  workload. 

Of  particular  concern  are  situations  having  sustained  mental 
workloads  of  long  duration.  Many  aircraft  missions  continue  to  require 
such  effort  by  the  crew.  But  the  introduction  of  computers  and 
automation  In  many  systems  has  come  to  mean  that  for  long  periods  of  time 
operators  have  nothing  to  do— the  workload  may  be  so  low  as  to  result  in 
boredom  and  serious  decrement  in  alertness.  The  operator  may  then 
suddenly  be  expected  to  observe  events  on  a  display  and  make  critical 
judgments— Indeed,  even  to  detect  an  abnormality,  diagnose  what  failed, 
and  take  over  control  from  the  automatic  system.  One  concern  Is  that  the 
operator,  not  being  "In  the  loop,**  will  not  have  kept  up  with  what  Is 
going  on,  and  will  reed  time  to  reacquire  that  knowledge  and  orientation 
to  make  the  proper  diagnoses  or  take  over  control.  Also  of  concern  Is 
that  at  the  beginning  of  the  transient  the  computer-based  Information 
will  be  opaque  to  the  operator,  and  It  will  take  some  time  even  to  figure 
out  how  to  access  and  retrieve  from  the  system  the  needed  Information. 

There  have  been  three  approaches  to  measuring  mental  workload.  One 
approach,  used  by  the  aircraft  manufacturers,  avoids  coping  directly  with 
measurements  of  the  operator  per  se  and  bases  workload  on  a  task 
tlme-llne  analysis:  the  more  tasks  the  operator  has  to  do  per  unit  of 
time,  the  greater  the  workload.  This  provides  a  relative  Index  of 
workload  that  characterises  task  demand,  other  factors  being  equal.  It 
says  nothing  about  the  mental  workload  of  any  actual  person  and  Indeed 
could  apply  to  a  task  performed  by  a  robot. 
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The  second  approach  la  perhaps  the  simplest — to  use  the  operator's 
subjective  ratings  of  his  or  her  perceived  mental  workload*  This  may  be 
done  during  or  after  the  events  judged.  One  form  of  this  Is  a 
single-category  scale  similar  to  the  Cooper-Harper  scale  for  rating 
aircraft  handling  quality.  Perhaps  more  interesting  Is  a  three-attribute 
scale,  there  being  some  consensus  that  "fraction  of  total  time  busy," 
"cognitive  complexity,"  and  "emotional  stress"  are  rather  different 
characteristics  of  mental  workload  and  that  one  or  two  of  these  can  be 
large  when  the  other(s)  are  small.  These  scales  have  been  used  by  the 
military  services  as  well  as  aircraft  manufacturers.  A  criticism  c:  »nec 
is  that  people  are  not  always  good  judges  of  their  own  ability  to  perr&rs 
in  the  future.  Some  pilots  may  judge  themselves  to  be  quite  capable  of 
further  sustained  effort  at  a  higher  level  when  in  fact  they  are  not. 

The  third  approach  is  the  so-called  secondary  task  or  reserve 
capacity  technique.  In  it  a  pilot  or  operator  is  asked  to  allocate 
whatever  attention  is  left  over  from  the  primary  task  to  Borne  secondary 
task,  such  as  verbally  generating  random  numbers,  tracking  a  dot  on  a 
screen  with  a  small  joy  stick,  etc.  Theoretically,  the  better  the 
performance  on  the  secondary  task,  the  less  the  time  required  and 
therefore  the  less  the  mental  workload  of  the  primary  task.  A  criticism 
of  this  technique  is  that  it  is  intrusive;  it  may  Itself  reduce  the 
attention  allocated  to  the  primary  task  and  therefore  be  a 
self-contaminating  measure.  And,  in  real  flight  operations  the  crew  may 
not  be  so  cooperative  in  performing  secondary  tasks. 

The  fourth  and  final  technique  is  really  a  whole  category  of 
partially  explored  possibilities — the  use  of  physiological  measures. 

Many  such  measures  have  been  proposed,  including  changes  in  the 


electroencephalogram  (ongoing  or  ateady-state),  evoked  reaponae 
potentials  (the  best  candidate  Is  the  attenuation  and  latency  of  the 

so-called  P^qq,  occurring  300  milliseconds  after  the  onset  of  a 
challenging  stimulus),  heart  rate  variability,  galvanic  skin  response, 
pupillary  diameter,  and  frequency  spectrum  of  the  voice.  All  of  these 
have  proved  to  be  noisy  and  unreliable. 

Both  the  Air  Force  and  the  Federal  Aviation  Administration  currently 
have  major  programs  to  develop  workload  measurement  techniques  for 
aircraft  piloting  and  traffic  control. 

If  an  operator's  mental  workload  appears  to  be  excessive,  there  are 
several  avenues  for  reducing  It  or  compensating  for  It.  Flrat,  one 
should  examine  the  situation  for  causal  factors  that  could  be  redesigned 
to  be  quicker,  easier,  or  less  anxiety-producing.  Or  perhaps  parts  of 
the  task  could  be  reassigned  to  others  who  are  less  loaded,  or  the 
procedure  could  be  altered  so  as  to  stretch  out  In  time  the  succession  of 
events  loading  the  particular  operator.  Finally,  It  may  be  possible  to 
give  all  or  part  of  the  task  to  a  computer  or  automatic  system. 

Cognitive  Science 

It  Is  Important,  for  purposes  of  evaluating  both  mental  workload  and 
cognitive  models  as  discussed  In  the  previous  section,  to  note  that  there 
has  been  an  enormous  change  In  models  of  mental  processing  In  both 
psychology  and  computer  science.  In  their  recent  paper,  Feldman  and 
Ballard  (In  press)  argue  that 
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Contemporary  computer  science  haa  sharpened  our  notions  of  what 
Is  "computable1*  to  Include  bounds  on  time,  storage  and  other 
resources.  It  does  not  seem  unreasonable  to  require  that 
computational  models  and  cognitive  science  be  at  least  as 
plausible  in  their  postulated  resource  requirement. 

The  critical  resource  that  Is  most  obvious  is  time.  Neurons, 
whose  basic  computational  speed  is  a  few  milliseconds  must  be 
made  to  account  for  complex  behaviors  which  are  carried  ouc  ir.  a 
few  hundred  milliseconds  .  .  •  (Posner,  1978).  This  means  that 
higher  complex  behaviors  are  carried  out  In  less  than  a  hunched 
time  steps.  It  may  appear  that  the  problem  posed  here  is 
Inherently  unsolvable  and  that  we  have  made  an  error  in  our 
formulation,  but  recent  results  In  computational  complexity 
theory  suggest  that  networks  of  active  computing  elements  can 
carry  out  at  least  simple  computations  In  the  required  time 
range — these  solutions  Involve  using  massive  numbers  of  units 
and  connections  and  we  also  address  the  question  of  limitations 
on  these  resources. 

There  Is  also  evidence  from  experimental  psychology  (Posner,  1978) 
that  the  human  mind  Is,  at  least  In  part,  a  parallel  system.  From 
neuropsychological  considerations  there  Is  reason  to  suppose  that  a 
parallelism  is  represented  In  regional  areas  of  the  brain  responsible  for 
different  sorts  of  cognitive  functions.  For  example,  we  know  that 
different  visual  maps  (Cowey,  1979)  underlie  object  recognition  and  that 
separate  portions  of  the  cortex  are  Involved  In  the  comprehension  and 
production  of  language.  We  also  know  more  about  the  role  of  subcortical 
and  cortical  structures  In  motor  control. 
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The  study  of  mental  workload  has  simply  not  kept  up  with  these 
advances  in  the  conceptualization  of  the  human  mind  as  a  complex  of 
subsystems.  The  majority  of  researchers  of  human  workload  have  studied 
tlie  interference  of  one  complex  task  with  another.  There  is  abundant 
evidence  in  the  literature  that  such  Interference  does  occur.  However, 
this  general  Interference  may  account  for  only  a  small  part  of  the 
variance  in  total  workload.  More  Important  may  be  the  effects  of  the 
specific  cognitive  systems  shared  by  two  tasks.  Indeed,  Kln6bourne  and 
Hicks  (1978)  have  recently  formulated  a  theory  of  attention  in  which  the 
degree  of  facilitation  or  Interference  between  tasks  depends  on  the 
distance  between  their  cortical  representation.  The  notion  of  distance 
may  be  merely  metaphorical,  since  we  do  not  know  whether  it  represents 
the  actual  physical  distance  on  the  cortex  or  whether  It  Involves  a 
relative  Interconnectivity  of  cortical  ar*>a;  the  latter  idea  seems  more 
reasonable. 

Viewing  humans  In  terms  of  cognitive  subsystems  changes  the 
perspective  on  mental  workload  (see  Nsvon  and  Gopher,  1979).  It  Is 
unusual  for  any  human  task  to  Involve  only  a  single  cognitive  system  or 
to  occur  at  any  fixed  location  in  the  brain.  Most  tasks  differ  in 
sensory  modality,  in  central  analysis  systems,  and  In  motor  output 
systems.  There  is  need  for  basic  research  to  understand  more  about  the 
separability  and  coordination  of  such  cognitive  systems.  Ve  also  need  a 
task  analysis  that  takes  advantage  of  the  new  cognitive  systems  approach 
to  ask  how  tasks  distribute  themselves  among  different  cognitive  systems 
and  when  performance  of  different  tasks  may  draw  on  the  same  cognitive 
system.  There  is  also  an  obvious  connection  between  a  cognitive  systems 
approach  and  analysis  of  .individual  differences  based  on  psychomet.:. •:  or 
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Information  processing  concepts,  and  much  needs  to  be  done  to  link 
analysis  of  Individual  abilities  to  the  ability  to  time-share  activity 
within  the  sane  cognitive  system  or  across  different  systems  (Landman  and 
Hunt,  1982). 

An  emphasis  on  separable  cognitive  systems  does  not  necessarily  mean 
that  a  more  unified  central  controlling  system  is  unnecessary.  Indeed, 
widespread  interference  between  tasks  of  very  different  types  (Pc.’;.\tr , 
1980)  suggests  that  such  a  central  controller  is  a  necessary  aspect  of 
human  performance.  There  are  a  number  of  theoretical  views  addrerri 
the  problem  of  self-regulation  of  behavior,  particularly  in  strci_ru_ 
situations,  two  principles  have  been  applied  by  human  factors 
engineers:  The  first  is  that  attention  narrows  under  stress.  Thus,  more 
attention  is  allocated  to  central  aspects  of  the  tnsk  while  less 
attention  is  allocated  to  more  peripheral  or  secondary  aspects. 

Sometimes  this  principle  has  been  applied  to  positions  in  visual  space, 
arguing  that  peripheral  vision  is  sacrlfled  more  than  central  vision 
under  stress.  The  degree  to  which  the  general  principle  applies 
automatically  to  positions  in  visual  space  or  to  allocation  of  function 
within  tasks  is  simply  not  very  well  understood — but  it  should  be.  A 
second  principle  of  the  relationship  between  stress  and  attention 
suggests  that  under  stress  habitual  behaviors  take  precedence  over  new  or 
novel  behaviors.  The  idea  is  that  behaviors  originally  learned  under 
stressful  conditions  tend  to  return  when  conditions  are  again  stressful. 
This  view  is  particularly  important  with  respect  to  the  process  of 
changing  people  from  one  task  layout  to  another.  If  the  original 
learning  takes  place  under  high  stress  conditions  while  transition  occurs 
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under  relatively  low  stress  conditions,  a  stressful  situation  may  tend  to 
reinstate  the  responses  learned  in  the  original  configuration. 

Recently  cognitive  psychologists  have  Itegun  to  take  into  account 
emotional  responses  produced  under  conditions  of  stress  (Bower,  1981). 

One  development  emphasises  links  between  individual  differences  in 
emotional  responding  and  attention  (see  Posner  and  Rothbart,  1980,  for  a 
review).  Although  it  is  a  highly  speculative  hypothesis  at  this  time, 
this  work  suggests  that  attention  may  be  viewed  as  a  method  for 
controlling  the  decree  of  emotional  responding  that  occurs  during 
stressful  conditions.  In  particular,  differences  in  personality  and 
temperament  may  affect  the  degree  to  which  attention  and  other  mechanisms 
are  successful  in  managing  stress.  These  new  models  relate  emotional 
responding  to  more  cognitive  processes.  They  have  the  potential  of 
helping  us  understand  more  about  the  effects  of  emotion  and  how  it  may 
guide  cognition  and  behavior  under  stressful  conditions.  Since  this  work 
has  just  begun,  there  are  few  general  principles  to  link  the  emotional 
responses  to  cognition  as  yet.  Developments  along  this  line  could  be 
useful  for  human  factors  engineers,  particularly  those  Involved  in 
training  and  retraining  and  those  Involved  in  mangement  of  stress  under 
battlefield  conditions. 

For  the  most  part,  this  discussion  has  been  from  the  viewpoint  of  the 
overloaded  operator.  For  much  of  the  time,  however,  the  operator  may  be 
underloaded.  In  the  field  of  vigilance  research,  which  is  concerned  with 
human  behavior  in  systems  in  which  signal  detection  is  required  but  the 
signals  are  Infrequent  and  difficult  to  detect,  a  great  deal  is  known 
about  exactly  what  parameters  of  signal  presentation  affect  performance. 
The  signal  detection  model  (Green  and  Swets,  1966)  has  been  shown  to  be 
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useful  In  analyzing  such  behavior.  Again,  Its  applicability  has  not  been 
evaluated  In  more  complex  tasks  in  which  signals  are  represented  by  more 
complex  patterns  of  activity  as  would  be  the  case  in  supervisory  control 
systems  of  the  types  described  above. 

Human  Proficiency  and  Error:  Culpability,  Trust,  and  Ultimate  Authority 

Designers  of  the  large,  complex,  capital-intensive,  high-risk-of-failure 
systems  we  have  been  discussing  would  like  to  automate  human  operator, 
out  of  their  systems.  But  they  know  they  must  depend  on  them  to  plan, 
program,  monitor,  step  In  when  failureo  occur  with  the  automation,  and 
generalize  on  system  experience.  They  are  also  terrified  of  human  error. 

Both  the  commercial  aviation  and  the  nuclear  pcver  Industries  are 
actively  collecting  data  on  human  error  and  trying  to  use  it  analytically 
In  conjunction  with  data  on  failures  In  physical  components  and 
subsystems  to  predict  the  reliability  of  overall  systems.  The  public  and 
the  Congress,  in  a  sense,  are  demanding  it,  on  the  assumption  that  it  is 
clear  what  human  error  Is,  how  to  measure  It,  and  even  how  to  stop  it. 

Human  error  is  commonly  thought  of  as  a  mistake  of  action  or  judgment 
that  could  have  been  avoided  had  the  Individual  been  more  alert, 
attentive,  or  conscientious.  That  Is,  the  source  of  error  is  considered 
to  be  Internal  and  therefore  within  the  control  of  the  Individual  and  not 
Induced  by  external  factors  such  as  the  design  of  the  equipment,  the  task 
requirements,  or  lack  of  adequate  training. 

Some  behavioral  scientists  may  claim  that  people  err  because  they  are 
operating  "open  loop" — without  adequate  feedback  to  tell  them  when  they 
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•re  Is  error.  They  would  h«ve  supervisory  control  systems  designers 
provide  feedback  st  every  potentlsl  misstep.  Product  llsblllty  litigants 
sometimes  tske  s  more  extreme  stance— thst  equipment  should  be  designed 
do  that  It  is  error  proof,  without  the  opportunity  for  people  to  (begin 
to)  err,  get  feedback,  then  correct  themselves. 

The  concept  of  human  error  needs  to  be  examined.  The  assertion  that 
an  error  has  been  committed  Implies  a  sharp  and  agreed-upon  dividing  line 
between  right  and  wrong,  a  simple  binary  classification  that  is  obviously 
an  oversimplification.  Human  decision  and  action  Involve  a 
multidimensional  continuum  of  perceiving,  remembering,  planning,  even 
socially  interacting.  Clearly  the  fraction  of  errors  in  any  set  of  human 
response  data  is  a  function  of  where  the  boundry  is  drawn.  How  does  one 
decide  where  to  draw  the  line  dividing  right  from  wrong  across  the  many 
dimensions  of  behavior?  In  addition,  is  an  error  of  commission,  (e.g., 
actuating  a  switch  when  it  is  not  expected),  equivalent  to  an  error  of 
omission,  (e.g.,  falling  to  actuate  a  switch  when  it  is  expected)?  Is  it 
useful  to  say,  in  both  these  Instances,  an  error  has  been  committed? 

What  then  exactly  do  we  mean  by  human  error? 

People  tend  to  differ  from  machines  in  that  people  are  more  Inclined 
to  make  "common-mode  errors,"  in  which  one  failure  leads  to  another, 
presumably  because  of  concurrency  of  stimuli  or  responses  iu  space  or 
time.  Furthermore,  as  suggested  earlier,  if  a  person  is  well  practiced 
in  a  procedure  ABC,  and  must  occasslonally  do  DBE,  he  or  she  is  quite 
likely  in  the  latter  case  to  find  himself  or  herself  doing  DBC.  This 
type  of  error  is  well  documented  in  process  control,  in  which  many  and 
varied  procedures  are  followed.  X*.  addition,  when  people  are  under 
•tress  of  emergency  they  tend  more  often  to  err  (sometimes,  howevei , 


analyst*  may  assume  that  operators  are  aware  of  an  emergency  when  they 
are  not)*  People  are  also  able  to  discover  and  correct  their  own  errors, 
which  they  surely  do  in  many  large-scale  systems  to  avert  costly 
accidents. 

Presumably  the  rationale  for  defining  human  error  is  to  develop  means 
for  predicting  when  they  are  likely  to  occur  and  for  reducing  their 
frequency  (Swain  and  Gutman,  1980).  Various  taxonomies  of  human  error 
have  been  devised.  There  are  errors  of  omission  and  errors  of 
comisslcm.  Errors  may  be  associated  with  sensing,  memory,  decision 
making,  or  motor  skill.  Norman  (1981)  distinguishes  mistakes  (wrong 
intention)  from  slips  (correct  Intention  but  wrong  action).  But  at 
present  there  is  no  accepted  taxonomy  on  which  to  base  the  definition  of 
human  error,  nor  is  there  agreement  on  the  dimensions  of  behavior  that 
should  be  invoked  in  such  a  taxonomy. 

There  is  usefulness  in  both  a  case  study  approach  to  human  error  and 
in  the  accumulation  of  statistics  on  errors  that  lead  to  accidents.  Both 
these  approaches,  however,  require  that  the  Investigator  have  a  theory  or 
model  of  human  error  or  accident  causation  and  the  framework  from  which 
to  approach  the  analysis.  In  addition  there  is  a  need  to  understand  the 
causal  chain  between  human  error  and  accident. 

One  has  only  to  examine  a  sampling  of  currently  used  accident 
reporting  forms  to  realize  the  Importance  of  the  need  for  a  framework  for 
analyzing  human  error.  They  range  from  medical  history  forms  to 
equipment  failure  reports.  None  that  we  have  examined  deals 
satisfactorily  with  the  role  of  human  behavior  in  contributing  to  the 


accident  circumstances 
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Furthermore,  for  Occident  reports  to  be  useful,  their  aim  needs  to  be 
specified.  There  is  an  inherent  conflict  between  the  goals  of 
understanding  what  happened  and  attempting  to  fix  blame  for  it.  The 
former  requires  candor,  whereas  the  latter  discourages  it.  Other 
potential  biases  in  these  reports  include:  (a)  exaggerating  in  hindsight 
what  could  have  been  anticipated  in  foresight;  (b)  being  unable  to 
reconstruct  or  retrieve  hypotheses  about  what  was  happening  that  no 
longer  makes  sense  in  retrospect;  (c)  telescoping  the  sequence  of  events 
(making  their  temporal  course  seem  shorter  and  more  direct);  (d) 
exaggerating  one's  own  role  in  events;  (e)  falling  to  see  the  Internal 
logic  of  others'  actions  (from  their  own  perspective).  Variants  of  these 
reporting  biases  have  been  observed  elsewhere  (Nlsbett  and  Ross,  1980). 
Their  presence  and  virulence  in  accident  reports  on  supervisory  control 
systems  merits  attention. 

In  addition  to  these  fundamental  research  needs,  there  Is  a  variety 
of  related  Issues  particularly  relevant  to  supervisory  control  systems 
that  should  be  addressed. 

In  supervisory  control  systems  it  is  becoming  more  and  more  difficult 
to  establish  blame,  for  the  Information  exchange  between  operators  and 
computers  Is  complex,  and  the  "error,"  If  there  ever  was  any,  could  be  in 
hardware  or  software  design,  maintenance,  or  management. 

Most  of  us  think  we  observe  that  people  are  better  at  some  kinds  of 
tasks  than  computers,  and  computers  are  better  at  some  others. 

Therefore,  It  seems  that  It  would  be  quite  clear  how  roles  should  be 
allocated  between  people  and  computers.  But  the  interactions  are  often 
so  subtle  as  to  elude  understanding.  It  is  also  conventional  wisdom  to 
say  that  people  should  have  the  ultimate  authority  over  machines.  But 
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again,  in  actual  operating  systems  we  usually  find  ou*;  selves  ill  prepared 
to  assert  which  should  have  authority  under  what  circumstances  and  for 
how  long* 

Operators  in  such  systems  usually  receive  fairly  elaborate  training 
in  both  theory  and  operating  skills.  The  latter  is  or  should  be  done  on 
simulators,  since  in  actual  systems  the  most  important  (critical)  events 
for  which  the  operator  needs  training  seldom  occur.  Unfortunately  there 
has  been  a  tendency  to  standardise  the  emergencies  (classic  stall  or 
engine  fire  In  aircraft,  large-break  loss-of-cooling  accident  in  nuclear 
plants)  and  repeat  them  on  the  simulator  until  they  become  fixed  paLiorns 
of  response.  There  seldom  is  emphasis  on  responding  to  new,  unusual 
emergencies,  failures  in  combination,  etc.,  vhlch  the  rule  book  never 
anticipated.  Simulators  would  be  especially  good  for  such  training. 

A  frustrating,  and  perhaps  paradoxical,  feature  of  "emergency" 
Intervention  is  that  supervisors  must  still  rely  on  and  work  with  systems 
that  they  do  not  entirely  trust.  The  nature  and  success  of  their 
intervention  is  likely  to  depend  on  their  appraisal  of  which  aspects  of 
the  system  are  still  reliable.  Research  might  help  predict  what  doubts 
about  related  malfunctions  are  and  are  not  aroused  by  a  particular 
malfunction.  Does  the  spread  of  suspicion  follow  the  operator's  mental 
model  (e.g.,  lead  to  other  mechanically  connected  subsystems)  or  along  a 
more  associative  line  (e.g.,  mistrust  all  dials)?  A  related  problem  is 
how  experience  with  one  malfunction  of  a  complex  system  cues  the 
interpretation  of  subsequent  malfunctions.  Is  the  threshold  of  mistrust 
lowered?  Is  there  an  unjustified  assumption  that  the  same  problem  is 
repeating  itself,  or  that  the  same  information-searching  procedures  are 
needed?  How  is  the  expectation  of  successful  coping  affected?  Do 
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operator*  assume  that  they  will  have  the  same  amount  of  time  to  diagnose 
and  act?  Finally,  how  does  that  experience  generalise  to  other  technical 
systems?  Do  bad  experiences  lead  to  a  general  resistance  to  innovation? 

A  key  to  answering  these  questions  is  understanding  the  operators' 
own  attribution  processes.  Do  they  subscribe  to  the  same  definition  of 
human  error  as  do  those  who  evaluate  their  performance?  What  gives  them 
a  feeling  of  control?  How  do  they  assign  responsibility  for  successful 
and  unsuccessful  experiences?  Although  their  mental  models  should 
provide  some  answers  to  these  questions,  others  may  be  sought  in  general 
principles  of  causal  attribution  and  mlsattrlbutlon  (Harvey,  et  al., 
1976). 


CONCLUSIONS  AND  RECOMMENDATIONS 


Supervisory  control  of  large,  complex,  capital-intensive,  high-risk 
systems  Is  a  general  trend,  driven  both  by  new  technology  and  by  the 
belief  that  this  mode  of  control  will  provide  greater  efficiency  and 
reliability.  The  human  factors  aspects  of  supervisory  control  have  been 
neglected.  Without  further  research  they  may  well  become  the  bottleneck 
and  most  vulnerable  or  most  sensitive  aspect  of  these  systems.  Reseach 
is  needed  on: 

(1)  How  to  display  integrated  dynamic  system  relationships  in  a  way 
that  is  understandable  and  accessible.  This  Includes  how  best 
to  allow  the  computer  to  tell  the  operator  what  it  knows, 
assumes,  and  Intends. 
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(2)  How  boat  to  ollow  the  operator  to  toll  tho  computer  what  ho  or 
•ha  wants  and  why,  in  a  flexible  and  natural  way. 

(3)  How  to  discover  the  Internal  cognitive  model  of  the 
environmental  process  that  the  operator  is  controlling  and 
Improve  that  cognitive  representation  if  it  la  inappropriate. 

(4)  How  to  aid  the  cognitive  process  by  computer-based  knowledge 
structures  and  planning  models. 

(5)  Why  people  make  errors  in  system  operation,  how  to  minimize 
these  errors,  and  how  to  factor  human  errors  into  analyses  of 
system  reliability. 

(6)  How  mental  workload  affects  human  error  making  in  systems 
operation  and  refinement  and  standardization  of  definitions  and 
measures  of  mental  workload. 

(7)  Whether  human  operator  or  computer  should  have  authority  under 
what  circumstances. 

(8)  How  to  coordinate  the  efforts  of  the  different  humans  Involved 
in  supervisory  control  of  the  same  system. 

(9)  How  best  to  learn  from  experience  with  such  large,  complex, 
interactive  systems. 

(10)  How  to  Improve  communication  between  the  designers  and  operators 
of  technical  systems. 

Research  is  needed  to  Improve  our  understanding  of  human-computer 
collaboration  in  such  systems  and  on  how  to  characterize  it  in  models. 

The  validation  of  such  models  is  also  a  key  problem,  not  unlike  the 
problem  of  validating  socioeconomic  or  other  large-scale  system  models. 

In  view  of  the  scale  of  supervisory  control  systems,  closer 


collaboration  between  researchers  and  systems  designers  in  the 


development  of  ouch  oysteme  mey  be  the  beet  way  for  auch  reeearch, 
modeling,  and  validation  to  occur.  And  perhaps  data  collection  should  be 
built  in  to  the  normal — and  abnormal — operation  of  such  ayatems. 
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USER-COMPUTER  INTERACTION 


INTRODUCTION 

Electronic  computers  have  probably  had  a  more  profound  effect  on  our 
society,  on  our  ways  of  living,  and  on  our  ways  of  doing  business  than 
any  other  technological  creation  of  this  century.  Computers  help  manage 
our  finances,  checking  accounts,  and  charge  accounts.  They  help  schedule 
rail  and  air  travel,  book  theatre  tickets,  check  out  groceries,  diagnose 
illnesses,  teach  our  children,  and  amuse  us  with  sophisticated  games. 
Computers  make  It  possible  to  erase  time  and  distance  through 
telecommunications,  thereby  giving  us  the  freedom  to  choose  the  times  and 
places  at  which  we  work.  They  help  guide  planes,  direct  missiles,  guard 
our  shores,  and  plan  battle  strategies.  Computers  have  created  new 
Industries  and  have  spawned  new  forms  of  crime.  In  reality,  computers 
have  become  so  Intricately  woven  into  the  fabric  of  daily  life  that 
without  them  our  civilization  could  not  function  as  It  does  today.  Small 
wonder  that  all  these  effects  have  been  described  as  the  results  of  a 
computer  revolution. 

Gantz  and  Peacock  (1981)  estimate  that  the  total  computer  power 
available  to  U.S.  businesses  Increased  tenfold  in  the  last  decade,  and 
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chat  it  is  expected  to  double  every  two  to  four  years.  According  to  the 
most  recently  available  estimates  (U.S.  Bureau  of  the  Census,  1979), 
there  are  currently  about  IS  million  computers,  terminals,  and  electronic 
office  machines  in  the  United  States.  That  number  Is  expected  to  grow  to 
about  30-35  million  by  1985,  at  which  time  there  will  be  roughly  one 
computer-based  machine  for  every  three  persons  employed  in  the 
white-collar  work  force.  Spectacular  advances  in  computer  technology 
have  made  this  growth  possible,  decreasing  the  cost  of  computer  hardware 
at  the  rate  of  about  30  percent  a  year  during  the  past  few  decades 
(Dertouzos  and  Moses,  1980). 

Computers  are  still  not  as  widely  accepted  as  they  might  be.  T *  f 
study  by  Zoltan  and  Chapanis  (1982)  on  what  professionals  think  about 
computers,  over  500  certified  public  accountants,  lawyers,  pharmacists, 
and  physicians  In  the  Baltimore  area  filled  out  a  64-item  questionnaire 
on  their  experiences  with  and  attitudes  toward  electronic  computers.  SI*: 
factors  emerged  from  a  factor  analysis  of  the  data.  Factor  1,  the 
largest  in  terms  of  the  variance  accounted  for.  Is  a  highly  positive 
grouping  of  adjectives  attesting  to  the  competence  and  productivity  of 
computers,  such  as  efficient,  precise,  reliable,  dependable,  effective, 
and  fast.  Factor  II,  the  second  largest  in  terms  of  the  variance 
accounted  for,  is  made  up  of  highly  negative  adjectives:  dehumanizing, 
depersonalizing,  Impersonal,  cold,  and  unforgiving. 

Still  another  factor  in  the  Zoltan-Chapanis  study  Indicates 
discontent  with  computers  in  terms  of  their  ease  of  use.  The  respondents 
thought  that  computers  are  difficult  and  complicated  and  that  computing 
languages  are  not  simple  to  understand.  These  views  are  apparent  in 
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their  response*  to  such  statements  as:  "I  would  like  a  computer  to 
accept  ordinary  English  statements"  and  "1  would  like  a  computer  to 
accept  the  jargon  of  my  profession,"  both  of  which  they  agreed  with 
strongly. 

The  findiugs  of  that  study  are  generally  in  agreement  with  more 
Informal  reports  in  the  popular  press  and  other  media  about  difficulties 
people  have  with  computers  and  their  use.  Indeed,  concerns  about  making 
computers  easy  to  use  can  have  serious  economic  consequences  that  may 
have  to  be  faced  by  more  and  more  computer  manufacturers.  For  example,  a 
small  company  in  California  was  recently  awarded  a  verdict  for 
substantial  monetary  damages  because  of  the  Inadequate  performance  of  a 
computer  that  the  company  had  purchased  (Bigelow,  1981).  In  rendering 
his  opinion  substantiating  the  award,  the  presiding  judge  said,  "It's  a 
particularly  serious  problem,  it  seems  to  me,  in  the  computer  Industry, 
particularly  in  that  part  of  the  Industry  which  makes  computers  for 
flrst*tlme  users,  and  segks  to  expand  the  use  of  computers  by  .  .  . 
targeting  as  purchasers  businesses  that  have  never  used  computers  before, 
who  don't  have  any  experience  in  them,  and  who  don't  know  what  the 
consequences  are  of  a  defect  and  a  failure"  (Bigelow,  1981:94). 

In  Europe  resistance  to  computerisation  has  taken  a  somewhat 
different  form  than  that  in  the  United  States.  Television  programs 
roughly  equivalent  to  the  American  program  60  Minutes  have  been  broadcast 
about  the  real  and  imagined  evils  of  computers.  Several 
countries— Austria,  England,  France,  Germany,  and  Sweden  among  them — have 
prepared  strict  standards  for  the  design  of  computer  systems  and  have 
enacted  federal  laws  restricting  hours  of  work  at  computer  terminals. 
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Similar  regulations  may  soon  be  in  effect  in  this  country.  One 
difficulty  is  that  current  standards  and  regulations  about  computers  are 
sometimes  based  on  skimpy  and  unreliable  data  and  sometimes  on  no  data  at 
all  (Rupp,  1981).  Whatever  their  origins,  these  events  and  trends  are 
symptoms  of  fairly  widespread  uneasiness  and  malaise  about  computers, 
their  usefulness,  and  usability.  No  one  denies  that  computers  are  here 
to  stay.  The  important  question  is:  "Hot;  can  we  best  design  them  for 
effective  human  use?"  This  chapter  describes  some  of  the  research  needed 
to  answer  that  question. 

Research  needs  are  identified  throughout  the  chapter.  However 
desirable  it  might  appear  to  assign  specific  priorities  to  each,  wt  fed 
that  it  is  difficult  and  risky  to  do  so  for  at  least  three  reasons 
First,  computer  hardware,  software,  and  interface  design  features  are 
changing  very  rapidly  (for  a  summary  of  the  trends  and  progress  in 
computer  development  see  Branscomb,  1982).  So,  for  example,  the 
Increased  availability  of  modularly  arranged  components  for 
microcomputers  for  personal  use,  la  the  office  and  at  school  as  well  as 
new  networking  and  communications  features  allow  design  improvements  to 
be  made  quickly  by  trial  and  error.  As  Nickerson  (1969)  has  pointed  out, 
such  trial~and~error  design  improvements  can  be  made  more  quickly  than 
they  could  be  by  careful  laboratory  research  studies. 

Second,  practical  considerations  are  likely  to  be  significant 
determinants  of  what  research  can  be  performed.  Operational  computer 
systems  rarely  can  be  disrupted  for  research  purposes,  and  up-to-date 
hardware  and  software  as  well  as  appropriate  groups  of  users  are  not 
always  available.  Under  these  circumstances  it  takes  great  Ingenuity  to 


conduce  human  factors  research  on  user-computer  interactions  that  can 
produce  useful,  generalizable  results.  Constraints  and  opportunities  are 
therefore  more  likely  than  assigned  priorities  to  dictate  what  research 
is  performed. 

Third,  there  is  a  definite  need  for  good  human  factors  research  in 
all  the  areas  we  discuss,  even  with  the  caveat  that  technology  is 
changing  rapidly  and  good  research  is  difficult  to  conduct.  With  these 
qualifications  in  mind,  we  do  provide  at  certain  places  in  this  chapter, 
short  summaries  Indicating  those  research  needs  that  we  feel  have  higher 
priorities  than  others. 


.  THE  COMPUTER  SYSTEM 

Computer  systems  and  their  environments  have  been  diagrammed  and  modeled 
in  various  ways.  Figure  5-1  illustrates  elements  that  are  important  from 
a  human  factors  standpoint:  the  user,  the  task,  the  hardware,  the 
software,  the  procedures,  and  the  work  environment.  Together  they 
cluster  around  what  is  commonly  called  the  user-computer  Interface — that 
invisible  surface  that  binds  the  various  elements  together.  Diagramming 
a  computer  system  in  this  way  is  to  a  large  extent  artificial,  because 
tbe  various  elements  cannot  really  be  considered  in  Isolation.  As  will 
be  apparent  later  on,  there  are  interactions  among  all  of  them.  The 
figure  is  merely  a  convenient  way  of  structuring  and  organizing  the 
subtopics  of  this  chapter,  which  are  described  briefly  below  and  treated 
in  detail  in  subsequent  sections. 


1.  The  Users  Beginning  with  the  users  Is  e  natural  starting  point  for 


any  discussion  of  the  human  factors  involved  in  computer  systems. 

Focusing  on  users  Implies  what  is  sometimes  referred  to  as  user-oriented 
design,  rather  than  machine-oriented  design.  Perhaps  the  most  Important 
questions  about  users  are  “Who  exactly  are  the  users? "  “What  are  their 
characteristics?”  and  “How  can  user  requirements  be  translated  into 
design  requirements?" 

2.  The  Task  The  second  element  is  the  task  or  the  Job  that  the  user  has 
to  do  with  the  computer.  The  complexity  of  the  job,  the  kinds  of 
Information  the  operator  needs  to  perform  the  job,  and  the  constraints 

under  which  jobs  must  be  performed  are  all  relevant  considerations  in  the 

* 

human  factors  design  of  computer  systems.  Task  requirements  are 
discussed  in  the  section  on  users. 

3.  The  Hardware  Hardware  means  input  devices,  output  display,-  and 
signaling  devices,  and  the  work  station  that  the  computer  operator  has  to 
use. 

4.  The  Software  Software  generally  refers  to  the  data  bases,  computer 
programs,  and  procedures  available  id  &  computer  system. 

5.  Procedures  Procedures,  manuals,  and  documentation  are  often  Included 
under  software.  They  are  shown  separately  in  Figure  5-1  because  the 
problems  associated  with  manuals  and  documentation  are  somewhat  different 
from  those  associated  with  programming  languages,  commands,  and  menus. 


6.  The  Work  Environment  Generally  speaking,  computers  and  computer 


systems  are  found  in  relatively  benign  work  environments.  Nonetheless, 
some  features  of  the  work  environment — excessive  glare,  noise,  and 
sometimes  dirt  and  vibration— -have  to  be  considered  in  the  design  of  the 
user-computer  Interface.  Since  standard  human  factors  recommendations 
and  good  engineering  practice  are  usually  adequate  guides  for  designing 
most  work  environments  in  which  computers  are  located,  we  do  not  cover 
environmental  variables  in  this  chapter. 


USERS  AND  TASKS 


Computer  users  today  are  almost  as  varied  as  people  In  general.  Although 
there  have  been  a  number  of  attempts  to  categorize  or  classify  computer 
users  into  various  groups  or  along  various  dimensions,  there  Is  today  no 
generally  accepted  way  of  doing  either.  Computer  tasks,  by  contrast,  can 
be  classified  under  the  same  headings  as  are  used  in  task  analyses. 
Proceeding  from  the  more  global  to  the  more  detailed  they  are  jobs, 
functions,  tasks,  and  subtasks.  According  to  Ramsey  and  Atwood  (1979), 
most  of  the  literature  about  computer  tacks  Is  at  the  job  level.  Some 
people  think,  however,  that  computer  tasks  cannot  be  classified  in 
Isolation,  but  that  tasks  Interact  with  users  and  that  the  two  must  be 
treated  together.  Examples  are:  professional  programmers  designing 
systems,  professionals  using  application  programs  with  command  languages, 
occasional  users  using  application  programs  with  menus.  In  short, 
classifying  computer  users  and  tasks  Is  clearly  in  need  of  systematic 
work,  and  It  is  treated  more  fully  in  the  sections  that  follow.  We  rely 


In  our  discussion  on  the  exemplary  review  of  the  literature  on 
human-computer  interaction  by  Ramsey  and  Atwood  (1979),  which  was 
supported  by  the  Office  of  Naval  Research. 


Users 

Attempts  to  classify  users  have  followed  one  of  several  quite  different 
approaches.  The  first  Is  to  categorise  users  into  more-or-less  distinct 
groups  on  the  basis  of  their  familiarity  or  sophistication  with 
computers.  This  way  of  classifying  users  has  yielded  a  large  collection 
of  names.  Examples,  in  alphabetical  order,  are:  casual  users  (Martin, 

1973) ,  computer  professionals  (Barnard  et  al.,  1981),  dedicated  users 
(Martin,  1973),  discretionary  users  (Bennett,  1979),  experienced  users 
(Shackel,  1981),  familiar  users  (Ledgard  et  al.,  1981),  first-time  users 
(Al-Awar  et  al.,  1981),  the  general  public  (Shackel,  1981),  general  users 
(Miller  and  Thomas,  1977),  Inexperienced  users  (Dzlda  et  al.,  1978), 
naive  users  (Thompson,  1969),  noncomputer  specialists  (Shackel,  1981), 
nonprogrammers  (Martin,  1973),  occasional  users  (Hammond  et  al.,  1980), 
programmers  (Martin,  1973),  regular  users  (Dzlda  et  al.,  1978),  and 
untrained  users  (Martin,  1973). 

Another  way  of  categorizing  users  has  focused  more  on  the  nature  of 
the  user's  Job.  This  has  produced  such  categories  as:  analysts  (S.  L. 
Smith,  1981),  clerical  workers  (Stewart,  1974),  managers  (Eason,  1974), 
operators  (Smith,  1981),  programmers  (Martin,  1973),  rugged  operators 
(Martin,  1973);  service  personnel  (Smith,  1981),  specialists  (Stewart, 

1974) ,  and  technical  users  (Ramsey  and  Atwood,  1979). 
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Quite  e  different  way  of  classifying  users  Is  In  terms  of  underlying 
personal  characteristics*  Thus,  Ramsey  and  Atwood  suggest  obtaining  data 
about  users*  abilities,  acquired  skills,  general  background  (including 
formal  education),  sex,  age,  attitude  measures,  mechanical  (perhaps  also 
spatial)  aptitudes,  vocabulary  teat  performance,  recency  and  length  of 
training  periods,  training  scores,  cognitive  decision  style,  and  general 
intelligence. 

Another  classification  of  users*  characteristics  would  Include  data 
on  the  following: 

1.  Sensory  capacities,  e.g.,  visual  acuity 

2.  Motor  abilities,  e.g.,  typing  skills 

3.  Anthropometric  dimensions,  for  hardware  design 

4.  Intellectual  capacities,  e.g.,  general  Intelligence  and  special 
abilities  in  order  to  evaluate  reading  levels  for  information 
presented 

5.  Learned  cognitive  skills,  including  familiarity  with  the  English 
language 

6.  Mathematical  and  logical  skills 

7.  Experience  with  computers  and  proficiency  in  training 

8.  Personality,  e.g.,  attitudes  toward  computers 

Shneiderman  (1980),  by  contrast,  classifies  users  only  according  to  their 
semantic  and  syntactic  knowledge  about  computers.  This  way  of 
classifying  users  yields  the  simple  matrix  shown  in  Figure  5-2. 

The  diversity  of  approaches  that  have  been  taken  to  this  problem 
indicates  that  we  need  research  to  understand  and  identify  which  of  many 


Syntactic  Knowledge 


Semantic 

Knowledge 


little  a  lot 


little 

naive  user 

data  entry 
job  control 

— _ _J 

language  (JCL) 
novices 

a  lot 

Infrequent 

frequent 

* 

novice  user 

professional 

user 

•* 


FIGURE  5-2  Classification  of  Users  According  to  the  Extent  of  Their 
Semantic  and  Syntactic  Knowledge 


Source:  Adapted  from  Shneiderman  (1980) 
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possible  user  characteristics  are  important  for  software  design.  In 
addition,  research  Is  needed  to  understand  how  to  express  and  translate 
user  characteristics  into  terms  that  can  be  used  in  systems  design,  i.e., 
into  specifications  for  designers  of  system  software. 

It  is  important  to  recognize  that  all  users,  whether  they  are 
seasoned  systems  programmers  or  less  experienced  users,  continue  to  learn 
as  newer  systems  are  developed  and/or  updated.  For  that  reason.  Cuff 
(1980)  has  suggested  that  we  need  to  consider  the  casual  user  of 
computers  as  well  as  expert  or  naive  users.  Additional  dimensions  ox 
user  behaviors  could  give  us  evidence  of  the  functionality  of  systems, 
e.g.,  the  range  of  tasks  users  can  perform  with  a  given  system,  how  1^  „.g 
it  takes  a  user  to  learn  a  system  or  a  system  update,  and  the  time  it 
takes  a  user  to  perform  a  particular  task  or  job.  We  need  to  know  what 
kinds  of  errors  users  make  when  learning  new  systems  as  well  as  how  many 
errors  are  made  and  how  often  they  are  made  or  repeated,  how  well  users 
adapt  to  changes  in  system  software  (robustness)  that  are  "upward 
compatible,***  and  how  users  rate  subjectively  the  quality  of  the  output 
or  product  and  the  systems  that  perform  their  set  of  tasks. 

When  we  look  at  what  is  currently  known  about  the  novice  compared 
with  the  expert  user,  it  appears  that  the  former  is  generally  engaged  in 
problem  solving  and  is  very  susceptible  to  task-structure  variations. 


*Upward  compatible  means  that  commands  and  features  used  in  an  older 
version  of  software  are  still  available  in  a  newer  version,  although  the 
newer  version  may  provide  new  commands  or  features  that  are  more 
efficient  for  accomplishing  the  same  ends. 
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The  expert  systems  programmer  typically  interacts  with  a  computer  as  a 
routine  cognitive  skill  and  is  somewhat  immune  to  structural  variations 
in  the  tasks  performed  (see  Moran,  1981;  Mayer,  1981).  A  simple  dialog 
In  the  software  that  is  computer-initiated  and  tutorial  in  nature  is 
probably  more  appropriate  for  the  occasional  and  naive  user,  but  an 
abbreviated,  user-initiated  dialog  appears  to  be  more  appropriate  for  the 
experienced  user.  It  is  clear  that  we  need  to  gather  more  data  about 
problem-solving  strategies  and  preferences  across  different  types  of 
tasks  for  different  levels  of  users. 

Of  particular  concern  is  that  the  research  methods  used  in  evaluating 
user  characteristics  for  hardware  design  have  been  used  In  studies 
evaluating  user  characteristics  for  software  design.  It  la  not  known  if 
these  research  methods  are  appropriate  for  evaluating  software  use  or 
which  methods  will  provide  the  most  information  to  designers.  Moran 
(1981)  has  addressed  this  Issue  in  part. 

Perhaps  the  two  most  pressing  research  needs  in  this  erea-are  to  find 
some  meaningful  way  of  classifying  or  categorizing  users  and  translating 
user  characteristics  into  specific  recommendations  that  can  be  used  In 
the  design  of  computer  hardware,  software,  and  documentation. 


Tasks 

Most  computer  and  human  factors  specialists  agree  that  a  task  taxonomy  is 
needed  and  that  system  designers  need  a  set  of  benchmark  tasks  to 
evaluate  hardv&re/softvare  development  and  changes.  A  task  structure 
provides  the  rules  of  the  game  that  determine  the  rs.'.^e  of  actions  users 


can  and  cannot  taka  (Moran  1981).  Tasks  can  vary  In  several  ways.  They 
may  (1)  fulfill  different  functions  for  the  user,  e.g.,  professional, 
educational,  or  home  hobby  functions,  (2)  require  different  forms  of 
language  such  as  natural  language,  BASIC,  COBOL,  or,APL,  and  (3)  be 
performed  on  different  kinds  of  systems. 

In  addition,  almost  all  system  designers  recognize  that  the  user's 
Interface  with  a  computer  system  changes  as  tasks  or  jobs  change.  The 
user  interface  Includes  any  part  of  the  computer  system  that  the  user 
comes  in  contact  with  physically,  perceptually,  or  conceptually.  Taj 
user's  conceptual  model  of  the  system  to  be  used  to  perform  a  given  task 
is  part  of  that  Interface.  Thus,  we  also  need  research  to  understand  ■  ow 
to  discover  a  user's  conceptual  model (s)  when  he  or  she  is  interiac_.i& 
with  the  computer. 

Models  suggested  by  Moran  (1981)  Involve  explicit  information 
processes  that  spell  out  step-by-step  the  mental  operations  the  user  must 
go  through  to  complete  the  task  application.  These  models  need  to  be 
based  on  a  psychological  theory  of  users.  One  example  of  specific  models 
that  describe  individual  user  differences  in  understanding  calculator 
languages  is  described  by  Mayer  and  Bayman  (1981). 

It  would  be  helpful  if  a  subset  of  the  task  taxonomy  or  benchmark 
tasks  could  in  some  way  be  Integrated  Into  the  accounting  systems  of 
computers  so  that  system  designers  could  be  provided  with  statistical 
data  about  tasks  and  users.  These  statistics  on  users  should  Include 
information  about  the  user  type  and  systems  used  as  well  as  errors  in 
usage.  One  example  of  a  keystroke-level  model  for  evaluating  performance 
is  described  by  Card  et  al.  (1980). 
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Of  primary  need  are  systematic  studies  of  the  conceptual  models  of 
users  when  they  interact  with  a  variety  of  hardware  and  software  systems 
to  do  specified  sets  of  tasks,  e.g.,  text  editing,  numerical  problem 
solving,  or  querying  data  bases.  These  studies  should  choose  successful 
methodologies  for  producing  results  that  can  be  directly  applied  to 
system  design,  or  they  should  Include  new  methods  for  evaluating  the 
Interactions  of  user  characteristics  with  task  requirements.  Another 
pressing  problem  is  the  development  of  a  meaningful  task  taxonomy  that 
Includes  both  behavioral  and  cognitive  elements  for  a  set  of  four  or  five 
different  representative  tasks. 


COMPUTER  HARDWARE 

Computer  hardware  cannot  be  designed  in  isolation  because  the  kind  of 
hardware  available  on  a  computer  terminal  determines  In  part  the  kinds  of 
dialog  and  the  kinds  of  command  languages  that  can  be  Implemented  In  the 
system.  Ideally,  decisions  about  Important  aspects  of  computer  dialogs 
should  precede  decisions  about  terminal  hardware.  In  practice,  the 
reverse  often  occurs.  While  recognizing  that  these  interactions  exist 
and  that  they  are  Important  in  design,  we  discuss  the  human  factors 
aspects  of  computer  hardware  with  only  passing  reference  to  their 
software  implications. 
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Input  Devices 

Designers  of  Interactive  computer  ayateoa  can  aelect  from  a  very  large 
number  of  devices  for  inserting  information  into  computers.  Table  5-1, 
modified  from  the  work  of  Ramsey  and  Atwood  (1979),  lists  16  different 
kinds  of  input  devices,  comments  on  some  of  their  features,  and 
Identifies  the  principal  references  to  studies  of  these  devices.  Since 
the. situation  has  not  changed  materially  since  the  Ramsey-Atwood 
report  was  Issued,  its  findings  are  still  valid. 

By  far  most  of  the  work  on  computer  input  devices  has  been  done  on 
keyboards;  the  literature  is  large  and  varied.  Seibel's  chapter  in  the 
Van  Cott  and  Klnkade  (1972)  handbook  is  a  good  starting  point  for  anyone 
Interested  in  these  problems.  Ramsey  and  Atwood  reference  a  number  of 
studies  done  after  Seibel's  chapter  was  written,  and  there  is~a  fair 
amount  of  even  newer  work,  e.g.,  Hlrsch  (1981)  and  Hornsby  (1981).  The 
available  literature  on  keyboards  is  sufficient  to  answer  most  practical 
questions.  This  is  no  longer  an  area  urgently  in  need  of  extensive 
research. 

The  situation  with  regard  to  alternative  input  devices,  such  as  light 
pens,  touch  panels,  and  hand  printing,  is  different.  Most  of  the  work 
that  has  been  done  on  these  devices  has  compared  two  or  more  input 
devices  in  specific  applications.  There  are  not  many  studies  of  this 
kind  in  the  literature,  although  Card  et  al.  (1978)  did  evaluate  the 
speed  and  accuracy  of  four  devices  for  text  selection.  Research  is 
needed  that  will  lead  to  a  set  of  recommendations  ab^ut  the  kinds  of 
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.input  devices  that  arc  beat  suited  to  general  classes  of  tasks  (e.g., 
text  input,  input  of  numerical  data,  selection  of  commands  and  operands 
from  displays,  discrete  positional  {graphical]  input,  and  continuous 
positional  [graphical]  input)  and  perhaps  to  general  classes  of  work 
environments* 

A  much  more  serious  concern  is  that  there  have  been  practically  no 
studies  of  the  optimal  design  of  input  devices,  except  for  keyboards. 

That  is,  given  that  a  light  pen  is  better  than  a  keyboard  for  some 
applications,  hov  exactly  would  one  design  the  best  light  pen  for  tha 
job?  Research  is  clearly  needed  on  the  optimal  design  parameters  of  all 
input  devices  other  than  keyboards. 

Voice  input  to  computers  deserves  special  treatment  because  (1)  it 
does  not  Involve  a  physical  mechanism  that  the  user  manipulates  as  such 
and  (2)  speech  as  a  human  output  is  distinctly  different  from  the 
movements  of  fingers,  hands,  or  feet  that  are  required  for  the  activation 
of  most  conventional  computer  input  devices. 

Speech  has  a  number  of  characteristics  that  theoretically  make  it  an 
attractive  candidate  for  computer  inputs.  It  is  fast,  effective, 
versatile,  flexible,  and  requires  little  effort.  Moreover,  almost 
everyone  knows  how  to  talk,  so  that  training  is  generally  unnecessary. 

One  of  the  principal  reasons  why  speech  input  is  not  widely  used, 
however,  is  that  technology  has  not  been  able  to  provide  us  with  speech 
recognition  capabilities  that  even  begin  to  approximate  those  cf  human 
listeners.  Nonetheless,  the  state  of  the  art  is  advancing  rapidly. 

There  are  now  some  very  good  speech  recognition  devices  available  and 
their  capabilities  are  certain  to  increase  greatly  in  the  foreseeable 
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Although  speech  hat  aone  distinct  advantages  as  a  medium  of 
communication.  It  la  also  easy  to  Identify  applications  In  which  speech 
Input  to  computers  would  not  ba  desirable.  Some  of  these  applications 
Involve  certain  kinds  of  users  (for  example,  persona  with  speech 
Impediments),  others  the  task  (for  example.  Intricate  mathematical  and 
chemical  formulae  are  not  easily  described  orally),  and  still  others  the 
work  environment  (speech  Input  is  not  very  efficient  In  noisy 
environments).  For  more  reliable  guidance  about  applications  In  which 
the  voice  should  or  should  not  be  used,  the  only  source  of  help  are 
recommendations  comparing  visual  and  auditory  forms  of  presentation  (see 
Table  5-2). 

Table  5-2,  and  others  like  It  in  the  human  factors  literature  suffer 

from  four  major  defects.  .First,  Che  recommendations  are  oriented  more 

toward  output  devices  rather  than  input  devices — that  ia,  they  do  not 
/ 

compare  apaach  with  other  possible  forms  of  date  input.  However 
ettrectlve  speech  may  appear  as  an  input  medium,  some  date  are  available 
suggesting  that  it  is  not  necessarily  the  solution  for  ell  situations 
(see,  for  example,  Sraunstein  and  Anderson,  1961).  Second, 
recommendations  such  as  those  in  Table  5-2  are  not  specifically  oriented 
toward  computer  applications.  Third,  these  comparisons  are  not 
sufficiently  comprehensive  to  be  of  much  use  to  computer  designers.  For 
example,  none  of  these  comparisons  considers  in  detail  user 
characteristics  or  the  work  environment  In  which  computers  are  used. 

Some  environments  have  rows  and  rows  of  computer  terminals  in  close 
proximity.  Imagine  the  babble  that  might  result  if  50  operators  were 
inputting  information  by  voice  simultaneously  into  computers!  Finally, 
existing  comparisons  of  vision  and  audition  provide  information  that  is 
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TABLE  5-2  Recommendations  for  the  Uee  of  Auditory  end  Vleual  Forms  of 
Presentation 


Use  auditory  presentation  if: 

Use  visual  presentation  If: 

1.  The  meesage  is  simple. 

1.  The  message  Is  complex. 

2.  The  message  la  short. 

2.  The  message  Is  long. 

3.  The  meseege  will  not  be 
referred  to  later. 

3.  The  message  will  be  referred 
to  later. 

4.  The  message  deals  with  events 

In  time . 

4.  The  message  deals  with 
location  In  space. 

5.  The  message  calls  for  Immedi¬ 
ate  action. 

5.  The  message  does  not  call 
for  immediate  action. 

6.  The  visual  system  of  the  person 

Is  overburdened. 

6.  The  auditory  system  of  the 
person  is  overburdened. 

7 .  The  'deceiving  location  Is  too 
bright  or  dark-adaptation 
Integrity  Is  necessary. 

7.  The  receiving  location  Is 
too  noisy. 

8.  The  person's  job  requires 
continual  movement. 

8.  The  person's  job  allows  for 
a  stationary  position. 

Source:  Deatherage  (1972) 
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too  vague  to  be  of  any  practical  usa  to  a  computer  designer.  For 
example,  how  la  a  designer  to  decide  whether  a  message  is  simple  or 
complex? 

What  we  clearly  need  is  a  detailed,  comprehensive,  and  quantitative 
set  of  guidelines  about  the  precise  conditions  under  which  speech  input 
to  computers  is  and  is  not  desirable.  These  guidelines  should  consider 
the  user,  the  task,  and  the  work  environment  in  which  computers  are 
located. 

Although  some  very  good  speech  recognition  machines  are  available, 
they  have  some  Important  limitations.  First,  they  all  are  word 
recognition  devices,  that  is,  they  do  not  recognise  continuous  speech. 
Second,  they  are  capable  of  responding  only  to  vocabularies  of  restricted 
size.  Third,  they  are  user-dependent,  that  is,  they  must  be  programmed 
to  learn  to  recognize  words  spoken  by  a  particular  person  and  will 
generally  respond  accurately  only  to  that  person's  voice.  Speech 
recognition  machines  that  can  respond  to  connected  speech  or  that  «re 
speaker-independent  are  yell  beyond  the  current  atate  of  technology. 

Despite  these  Important  limitations,  speech  input  to  computers  can  be 
successful  and  useful.  There  is  not,  however,  a  good  base  of  research 
findings  on  the  conditions  under  which  speech  recognition  machines  can  be 
used  effectively  even  with  their  limitations.  For  example,  how  much 
useful  work  can  be  done  with  vocabularies  of  various  sizes?  Bow 
effectively  can  people  be  trained  to  leave  pauses  between  words  in 
connected  speech  so  that  individual  words  can  be  recognized?-  Bow 
effortful  is  it  to  speak  while  deliberately  leaving  pauses  between 
words?  If  vocabularies  of  restricted  size  must  be  used,  how  effectively 
can  one  construct  complex  Inputs  with  the  available  words?  What  rules  of 
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grammar  and  syntax  must  be  observed  if  one  is  restricted  to  a  limited 

vocabulary?  What  should  that  vocabulary  be?  The  conditions  under  which 

speech  recognition  devices  can  be  used  most  effectively  is  virtually  an 
* 

unexplored  area  of  research  that  should  be  vigorously  pursued.  One 
example  of  research  in  the  use  of  voice  input  to  operate  a  distributed 
computer  network  has  been  conducted  at  the  Navy  Postgraduate  School  by 
Poock  (1980). 

Output  Devices 

Although  teletypewriters  and  alphanumeric  cathode  ray  tube  (CRT)  display* 
are  the  most  common  forms  of  output  devices  used  in  computer  systems, 
there  are  numerous  other  possibilities:  plasma  displays;  light-emitting 
diodes  (LED)  and  liquid  crystal  displays;  tactile  displays;  audio 
displays,  including  synthetic  speech;  graphical  displays;  laser  displays; 
and  even  psychophyslologlcal  output  devices.  The  state  of  the  art  of 
these  various  output  dsvlces  is  summarized  in  Table  5-3,  which  is  based 
on  Ramsey  and  Atwood  (1979). 

t 

CRT  Displays 

Enough  research  has  been  done  on  CRT  displays  to  support  guidelines  for 
their  design  (Galitz,  1981;  Shurtleff,  1980).  Although  the  two  handbooks 
available  do  not  answer  all  the  questions  designers  may  have,  they  cover 
e  substantial  number  of  them.  Most  of  their  recommendations  are 
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Research  dealing  directly  with  the  line  printer  ueed  In  cc m-  (1967)* 

puter  ctetnat  le  scanty  but  consistent  with  findings  of  typo-  Levis  (3.972)** 
graphic  research  (e.g.,  nixed  upper-lover  cate  ie  beat  for  Ling  (1973) 
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supported  by  research  data,  and  those  that  are  not  seem  reasonable.  The 
two  oost  Important  unresolved  questions  concern  the  size  of  displays  and 
the  use  of  colored  displays. 

With  regard  to  else,  Shurtleff  (1980)  has  devoted  a  chapter  to 
questions  of  legibility  as  related  to  display  slse,  but  he  has  nothing  to 
say  about  the  more  Important  question  of  how  much  information  can  be 
presented  on  screens  of  various  sizes.  Military  applications  of  computer 
displays,  for  example,  In  cockpits,  must  be  small  by  necessity.  Bow 
small  c*»n  they  be  and  still  be  legible?  Bow  can  information  best  be 
presented  on  small  displays?  The  converse  problem  may  occur  when  many 
people  must  view  the  same  display.  In  that  case  the  relevant  questions 
are:  How  large  can  displays  be?  How  can  Information  best  be  presented 
on  large  displays?  These  are  not  questions  relating  simply  to  the 
legibility  of  the  Information  presented  on  displays  of  various  size;  such 
questions  can  easily  be  resolved  on  the  basis  of  available  data.  What  Is 
needed  Is  research  on  the  Interactions  between  display  size  and  the 
amount  of  Information  that  can  be  most  effectively  presented. 

Questions  on  the  use  of  color  on  CRT  displays  Is  also  still  . 
essentially  unresolved.  The  advantages  of  color  coding  for 
identification  purposes  are,  of  course,  well  documented,  but  the 
long-term  effects  of  working  with  colored  CRT  displays  for  data  entry. 
Inquiry,  or  interactive  dialog  are  not  known.  Although  many  people  seem 
to  like  colored  displays,  others  find  them  annoying  and  garish.  The 
scanty  research  evidence  available  seems  to  show  that  colored  CRT 
displays  produce  nr  substantial  performance  benefits.  More  research  may 
enable  designers  to  make  Informed  decisions  about  the  possible  benefits 
of  color  on  CRTs  versus  their  cost  and  other  disadvantages. 
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Alternative*  to  CRT  Display* 

* 

V#.ry  little  human  factor*  reaearch  ha*  been  done  on  dlaplay*  other  than 
CRT*.  Of  particular  interest  are  synthetic  speech  display*. 
Computer-generated  speech  la  now  available  in  a  variety  of  device*,  and 
the  quality  of  the  speech  in  some  of  these  devices  is  quite  good.  The 
situations  in  which  computer-generated  speech  Is  a  viable  alternative  to 
visual  displays,  however,  are  not  known  .  Basic  research  paralleling 
that  on  speech  input  is  needed  to  produce  defensible  recommendations 
about  applications  in  which  speech  output  can  or  should  be  used* 


Workplace  Design 

•  • 

Computer  displays  and  input  devices  are  generally  assembled  into  work 
stations  consisting  of.  terminals,  consoles,  desks,  and  chairs.  There  is, 
of  course,  a  very  large  and  useful  literature  on  the  physical  layout  of 
workplaces  (see,  for  example,  Van  Cott  and  Kinkade,  1972),  but  there  is 
very  little  empirical  research  on  work  station  design  specifically  for 
computor-related  tasks  and  settings.  The  importance  of  these  problems  is 
highlighted  by  a  great  deal  of  literature,  mostly  from  Europe,  about 

complaints  from  workers  using  CRT  devices  (see,  for  example,  Crahdjean 
•nd  Vigliani,  1980). 

Similar  complaints  from  a  consortium  of  labor  unions  in  the  United 
States  were  received  by  the  National  Institute  of  Occupational  Safety  and 


Health  (NIOSH)  in  1979*  The  general  nature  of  these  complaints  was  that 

employees  using  CRT  terminals  experienced  a  variety  of  symptoms  including 

headaches,  gensral  malaise,  eyestrain,  and  other  visual  and 
% 

musculoskeletal  problems.  In  response  to  these  complaints  NIOSH 
conducted  an  extensive  investigation  of  computer  work  stations  in  three 
companies  in  the  San  Francisco  Bay  area  (Murray  at  al.,  1981).  The  study 
consisted  of  four  phases:  (1)  radiation  measurements,  (2)  industrial 
hygiene  sampling,  (3)  a  survey  of  health  complaints  and  psychological 
mood  states,  and  (A)  ergonomics  and  human  factors  measurements. 

Although  radiation  from  CRTs  had  long  been  suspected  as  a  potential 
health  hazard,  the  NIOSH  study  seems  to  have  conclusively  ruled  it  out. 
X-ray,  ultraviolet,  and  radio-frequency  radiation  in  all  sites  and  at  all 
work  stations  tested  was  either  not  detectable  or  was  well  below 
acceptable  occupational  levels.  Similar  negative  conclusions  were 
reached  about  the  chemical  environment.  Hydrocarbon,  carbon  monoxide, 
acetic  add,  and  formaldehyde  levels  in  and  around  work  stations  were  not 
appreciably  different  from  what  one  would  find  In  an  ordinary  living 
environment • 

The  results  of  the  survey  of  health  complaints  were  quite  different, 
however.  They  show  that  operators  of  visual  display  terminals  (VDT) 
experienced  a  greater  number  of  health  complaints,  particularly  related 
to  emotional  and  gastrointestinal  problems,  than  did  comparable  operators 
who  did  not  work  with  VDTs.  These  findings,  according  to  the  NIOSH 
report,  demonstrate  a  level  of  emotional  distress  for  the  VDT  operators 
that  could  have  potential  long-term  health  consequences.  The  NIOSH  study 
concludes,  however,  that  it  is  quite  likely  that  the  emotional  distress 
shown  by  the  VDT  operators  is  more  related  to  the  type  of  work  activity 
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than  to  the  use  of  VDTs  per  se.  With  the  growing  number  of  VDTe  in  our 

society,  It  le  clearly  of  considerable  Importance  to  establish  how  much 

of  worker  complaints  can  be  traced  to  VDTs  and  how  much  to  other  factors 
* 

(Ketchel,  1981;  M.  J.  Smith,  1981).  This  Is  a  research  question  that 
urgently  needs  to  be  Investigated. 

The  NIOSH  report  has  more  to  say  about  the  ergonomic  and  human 
factors  aspects  of  the  computer  workplace  than  about  any  other  aspect  of 
computer  work.  Keyboard  heights,  table  and  chair  designs,  viewing 
distances  and  viewing  angles,  copy  holders,  and  other  aspects  of  work 
station  design  all  come  In  for  criticism.  Computer  work  stations  in 
America  appear  to  be  as  poorly  designed  as  those  In  Europe  (see  Grandjean 
and  Vigllanl,  1980;  Brown  et  al.,  1982),  forcing  operators  to  adopt 
strained  postures  and  to  contend  with  glare  and  generally  substandard 
viewing  conditions  (Ketchel,  1981).  Although  basic  data  for  good  work 
station  design  are  available,  they  need  to  be  assembled  In  a  good  set  of 
guidelines  specifically  oriented  toward  such  design.  This  also  appears 
to  be  an  urgent  research  need. 


General  Problems 

Three  general  problems  relating  to  computer  hardware  have  received  almost 
no  attention:  (1)  the  design  of  transportable  terminals  and  data,  (2) 
the  design  of  robust  computer  systems  for  military  purposes,  and  (3)  the 
design  of  computer  terminals  for  use  In  unusual  or  exotic  environments, 
for  example,  In  moving  vehicles  or  under  water. 


Spectacular  advancaa  in  nicroalactronlca  hava  mada  it  poaaibla  to 

package  enormous  computing  power  into  small  packages.  The  full  potential 

of  this  miniaturization  has  not  yst  been  realized  or  explored.  We  need 
* 

human  factors  research  leading  to  the  design  and  use  of  transportable 
terminals,  including  input  and  output  devices  and  data  in  the  form  of 

cassettes. 

Most  computer  systems  are  designed  for  use  in  benign  environments. 

As  the  use  of  computers  becomes  more  common  in  the  military  services, 
data  will  La  urgently  needed  on  how  to  design  them  for  the  rough 
treatment  they  are  almost  certain  to  receive  under  operational  conditions. 

Vibration,  high-g  forces,  immersion  in  water,  and  perhaps  other 
environmental  conditions  affect  machines  as  well  as  their  operators. 
Certain  input  devices,  for  example,  light  pens  or  even  keyboards,  may  be 
difficult  or  impossible  to  use  when  the  computer  and  the  operator  are 
subjected  to  excessive  movement,  vibration,  or  g~forces.  We  have 
essentially  no  information  about  the  usability  of  computers  or  the  design 
of  computers  for  use  under  such  conditions.  Although  this  may  not  be  an 
immediate  problem,  it  la  certain  to  become  increasingly  Important  as 
computers  are  Integrated  into  complex  systems  for  use  in  harsh,  exotic, 
or  unusual  environments. 


COMPUTER  SOFTWARE 


Software  has  many  different  meanings  to  computer  scientists  and  computer 
analysts  who  develop  or  use  computer  programs  that  include  command 
languages,  dialog  systems,  and  specialized  applications  systems  with  data 
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bases.  Software  nay  have  originally  been  synonymous  with  computer 
programs,  but  In  general  software  now  consists  of  "the  operational 
requirements  for  a  system,  its  specifications,  design,  and  programs,  all 
its  user  manuals  and  guides,  and  its  maintenance  documentation"  (Mills, 
1980:417). 

Research  in  human  factors  in  software  has  evaluated  the 
human-computer  Interface  with  command  languages,  programming  languages, 
dialog  systems,  and  feedback  and  error  management.  Frequently  the  human 
factors  studies  have  emphasized  ease  of  use  and  ease  of  learning  as  well 
as  efficiency  of  completing  the  problem-solving  tasks  on  the  computer. 

The  recent  experimental  and  observational  studies  were  summarized  in  the 
special  issue  on  human  factors  in  Computing  Surveys  (1981),  the  IBM 
Systems  Journal  (1981) ,  and  in  articles  in  Human  Factors,  the 
International  Journal  of  Man  Machine  Studies,  and  Ergonomics.  In 
addition,  there  are  exemplary  technical  reports,  such  as  Wllliges  and 
Wllliges  (1981),  Ledgard  et  al.  (1981),  Shneiderman  (1980),  and  the 
proceedings  of  the  Conference  on  Human  Factors  in  Computer  Systems 
(Institute  for  Computer  Sciences  and  Technology,  1982).  The  more  popular 
trade  magazines,  e.g.,  the  April  1982  Issue  of  BYTE,  also  feature 
articles  on  human  factors  in  software  design.  Many  authors  express  the 
need  for  additional  careful  research  studies  in  software  design  and 
criticize  many  current  results  as  incomplete  and  inconsistent  due  to  poor 
methodology,  use  of  subject  populations  limited  to  particular  types  of 
users  (e.g.,  college  students).  Inadequate  experimental  designs,'  and 
misuse  or  poor  use  of  statistics. 

Selected  useful  guidelines  for  software  designers  are  found  in  Engle 
and  Granda  (1975)  and  the  recent  reports  by  Willigec  and  Wllliges  (1981) 
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and  Ehrenreich  (1981).  Although  there  exist  guidelines  as  wsll  as 

selected  research  studies  Id  human  factors  Issues  Id  software, 

considerable  research  needs  to  be  done  in  order  to  provide  information  of 
% 

use  to  system  designers  of  software. 

•  The  research  efforts  needed  in  human  factors  in  software  design  can 

*  .  i 

be  divided  into  two  areas:  (1)  methodological  studies  and  (2) 
substantive  studies  of  software  design  features  for  the  end  user.  The 
two  areas  are  not  always  independent,  and  some  research  atudies  require 
attention  to  both.  In  either  case  we  are  concerned  about  human  factors 
research  in  software  eye terns  with  which  end  users  interact  or  interface, 
not  about  research  in  programming  language  design  per  se;  this  is  usually 
the  concern  of  the  computer  programmer  or  systems  analyst. 

In  the  methodological  area,  research  is  needed  on  how  to  develop  a 
suitable  simulation  capability  for  the  design  of  dialog  and  interface 
systems.  We  need  to  understand  how  to  evaluate  present  software  systems 
as  well  as  how  to  mock  up  new  systems  for  testing  and  evalvrtica  with  end 
users.  The  choice  of  dependent  variables  in  evaluating  software  is  not 
clear.  We  know  little  about  how  to  collect  user  statistics  on  the  ease 
of  learning  of  new  software,  how  to  record  errors  and  complex 
response-time  metrics  from  end  users  in  time* sharing  systems,  and  how  to 
measure  user  satisfaction.  Research  is  needed  on  what  components  of 
usability  are  most  important  for  different  kinds  of  users  and 
applications  (see  Shackel,  1981). 

One  of  the  problems  in  this  area  is  that  we  don't  know  how  to  do 
research  on  these  topics.  There  is  no  agreed-upon  set  of  empirical 
methodologies  for  conducting  research  studies  about  software  issues.  The 
studies  that  have  been  done  are  frequently  context-specific  and/or  about 


37 


one  or  two  software  feature*  and  ara  difficult  to  generalize  and 
Integrate  with  other  data  In  the  area.  Examples  Include  evaluations  of  a 
given  command  asking  users  to  translate  the  abbreviated  form  into 
English,  effects  of  modifications  of  conditional  nesting  structures  In 
FORTRAN,  user  efficiency  of  Indentations  to  locate  single  bugs  in  PASCAL, 
and  modifications  In  a  language  used  In  teaching  at  the  University  of 
Toronto.  A  research  program  undertaken  by  a  multidisciplinary  group  at 
Virginia  Polytechnic  Institute  and  State  University  by  Wllllges  and 
Ehrich  sponsored  by  the  Office  of  Naval  Research  [human-computer 
Interaction  and  decision  behavior,  NR  SR0-101]  is  attempting  to  develop 
principles  of  effective  human-computer  Interaction,  including 
establishment  of  a  user's  model  of  command  languages.  This  research  is 
Interdisciplinary  and  programmatic  in  nature.  Another  set  of 
methodological  studies  Is  needed  to  discover  how  to  develop  guidelines 
and  what  kiads  of  guidelines  for  software  characteristics  are  most  useful 
for  system  designers  and  engineers;  for  example.  Smith  has  described  his 
ileas  and  progress  In  this  area  In  the  proceedings  of  the  Conference  on 
Human  Factors  in  Computing  Systems  (Institute  for  Computer  Sciences  and 
Technology,  1982). 

In  a  substantive  area,  research  is  needed  to  understand  the  control 
of  users'  input  accuracy  through  "clever"  or  "novel"  feedback  during 
actual  use;  experiences  as  well  as  what  the  "format  structures"  should  be 
for  providing  feedback  on  errors  that  users  make.  Data  needs  to  be 
collected  on  how  best  to  provide  effective  error  correction  features, 
help  messages,  and  what  range  of  default  procedures  should  be  provided  to 
aid  user  efficiency.  We  need  research  to  evaluate  how  important  feedback 
and  system  response  time  are  for  Improving  user  efficiency  or  ease  of 
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use.  There  is  a  need  for  methodology  and  quantification  of  user  ease  and 
efficiency.  At  present,  studies  evaluate  different  types  of  commands  In 
a  laboratory  rather  than  in  real-use  settings,  and  It  is  not  clear  that 
the  most  effective  commands  in  the  laboratory  are  applicable  in  applied 
system  uses.  We  need  information  on  what  length  of  commands  (one,  two, 
or  three  words)  or  how  many  (enter  only  one  and  wait  for  system  response 
or  enter  six  at  once)  are  preferred  by  casual  users  rather  than  expert 
software  programmers. 

A  variety  of  studies  are  needed  in  order  to  evaluate  how  best  to 
develop  natural  language  dialog  systems  and  in  particular  what  kinds  of 
language-based  models  of  human  communication  are  most  appropriate  for 
commands  in  operating  systems,  editing  systems,  knowledge-based  systems, 
and  query  systems  for  human  computer  interactions  (e.g.,  Reisner,  1981). 

Additional  reaeach  la  needed  to  understand  how  to  develop 
knowledge-based  systems  for  a  variety  of  users.  Knowledge-based  systems 
are  developed  by  a  formulation  of  the  application  problem,  designing  and 
constructing  the  knowledge  base  of  expertise,  developing  schemes  of 
Inference,  search,  or  problem  solving,  winning  the  confidence  of  experts, 
and  evaluating  the  programs  for  production  versions.  Examples  of 
knowledge-based  systems,  frequently  referred  to  as  expert  systems. 

Include  assisting  users  in  such  tasks  as:  (1)  deducing  molecular 
structures  from  the  output  of  mass  spectrometers,  (2)  advising  when  and 
where  to  drill  for  ore,  and  (3)  diagnosing  blood  infections.  It  should  . 
be  noted  that  there  are  three  different  kinds  of  end  users  of  these 
systems,  only  the  first  of  which  is  a  user  in  a  conventional  Information 
retrieval  system:  (1)  in  getting  answers  to  problems,  the  user  as 
client,  (2)  in  improving  the  system’s  knowledge,  the  user  as  a  tutor,  and 
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(3)  in  harvesting  the  knowledge  base,  the  user  as  pupil.  A  summary  of 

recent  research  related  to  knowledge-based  or  expert  systems  can  be  found 

in  L.  C.  Smith  (1980).  Some  of  the  major  features  of  these  systems, 

* 

including  the  schemes  of  inference  or  problem-solving  approaches  used  in 
defining  structures  for  the  knowledge  bases,  are  reviewed  by  Feigenbaum 
(1978). 

A  recently  developed  specialty  is  software  associated  with  special 
graphics  displays.  At  present  the  development  of  both  hardware  and 
software  for  graphics  use  are  at  the  gadget  stage.  We  need  to  know  how 
to  design  software  modules  for  graphics  use,  what  modules  are  best  for 
various  graphics  features  in  addition  to  points,  lines,  and  circles,  and 
how  to  mix  keyboard  and  pen  inputs  in  ways  other  than  up  and  down  arrows 
and  drawing  pad  devices.  Most  graphic  software  has  hierarchical  levels 
for  command  use;  it  is  unknown  if  different  levels  are  needed  or  how  many 
are  needed  and  which  commands  are  best  to  use  at  each  level.  Also,  the 
best  ways  for  interacting  among  the  hierarchically  ordered  levels  of 
commands  for  draw  and  edit  and  the  method  for  terminating  are  unknown* 

We  need  more  information  about  what  icons,  menus,  and  special  symbols 
should  be  used  in  creating  graphics.  Methods  have  been  developed  for 
partitioning  a  display  screen  into  multiple,  sometimes  overlapping 
windows,  each  monitoring  an  Independent  process.  There  has  been  very 
little  research  on  how  best  to  make  use  of  this  kind  of  capability.  We 
know  little  about  how  to  use  color  effectively  for  different  kinds  of 
graphics  displays  and  applications. 

i 

Several  of  the  above  research  recommendations  have  been  recognized  by 
Moran  (1981),  who  also  suggests  that  further  research  is  needed  to 
understand  users'  conceptual  models  in  Interacting  with  a  variety  of 
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software  systems.  In  addition,  Thomas  and  Carroll  (1981)  and  Miller 

(1981)  have  emphasized  that  the  areas  of  most  needed  research  are  in  the 

human-to-computer  communication  process,  Including  research  on  the 
*> 

advantages  and  disadvantages  of  natural  language  software  systems  for 
different  tasks.  Computers  have  become  more  a  part  of  all  office  systems 
today,  and  we  need  to  study  what  impact  the  new  computer  technology  has 
on  organizations  and  their  structures  as  well  as  the  effects  on  decision 
making  of  the  new  management  information  systems  (Federico,  1980). 

As  a  final  point.  It  should  be  noted  that  we  need  research  on  the 
Interaction  between  hardware  and  software  design  features  as  new 
developments  such  as  voice  input  and  video  disks  become  more  commonly 
incorporated  into  all  types  of  computer  systems. 

Important  research  that  should  be  done  involves  first  the  design  and 
analysis  of  new  methodologies  for  conducting  software  research,  anil 
second,  users'  conceptual  models  of  software  systems,  including  natural 
language  systems  for  a  variety  of  tasks.  Also,  we  need  to  understand  how 
to  develop  and  evaluate  additional  knowledge-based  systems  for  users  as 
client,  tutor,  and/or  pupil.  Also  needed  are  studies  conducted  to 
understand  what  software  features  would  facilitate  effective  use  of 
graphics  in  different  tasks. 


DOCUMENTATION 


Documentation  was  once  defined  as  printed  matter  that  describes  or 
explains  how  a  system  of  some  kind  works  or  should  be  used.  The 
documentation  was  necessarily  separate  from  the  system  unless  the  system 
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itself  was  a  thing  of  print  on  paper.  In  the  context  of  the  computer, 

however,  documentation  can  be  part  and  parcel  of  the  system  it  describes 

or  explains.  Recent  experience  indicates  that  on-line  documentation  has 
* 

many  advantages  over  print-on-paper  documentation,  ,1c  cannot  get  lost  or 
separated  from  the  system.  Inasmuch  as  the  user  is  working  with  the 
computer,  the  computer  can  monitor  what  the  user  is  doing  and  help  find 
the  parts  of  the  documentation  that  are  pertinent  to  the  user's  current 
activity  and  current  quandary.  When  the  user  thinks  he  or  she 
understands  what  to  do  the  computer  can  help  do  it — and  may  be  able  to 
try  it  out  in  a  tentative  way  that  will  not  cause  much  trouble  If  the 
user's  understanding  la  faulty.  The  possibilities  are  obviously 
revolutionary.  Because  on-line  documentation  is  relatively  new,  however, 
not  much  is  known  about  how  to  design  and  implement  It  effectively. 
Clearly  the  first  priority  for  research  in  documentation  Is  to  explore, 
evaluate,  and  improve  techniques  of  on-line  documentation. 

On-line  documentation  within  the  system  is  not  the  ansver..to  all 
needs  for  documentation,  of  course.  Some  computer  systems  (such  as 
batch-processing  systems  and  automatic  process-control  systems)  are 
noninteractlve,  and  others  (such  as  many  avionics  systems)  do  not  have 
enough  memory  or  storage  to  make  on-line  documentation  feasible. 
Documentation  for  such  systems  is,  by  and  large,  not  very  satisfactory. 
There  la  still  need,  therefore,  for  Improved  external  documentation, 
documentation  that  is  associated  with  the  system  but  not  in  it.  Wright 
(1981)  has  several  useful  suggestions  for  documentation  designer's, 
including  suggested  aids  that  take  the  form  of  heuristics  for  analyzing 
the  user's  interaction  with  the  text.  Her  suggestions  also  consider 
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types  of  users  and  Che  user's  (reader's)  purpose  rather  than  the  producer 
designer's  (writer’s)  purpose  as  a  classification  for  documents. 

Of  course,  external  documentation  need  not  necessarily  be 
print-on-paper  documentation.  It  is  an  interesting  idea  to  associate  a 
"documentation  computer"  with  the  system  to  which  the  documentation 
pertains.  In  some  Instances,  the  documentation  computer  might  be  a  small 
machine,  even  a  portable  one,  taking  the  place  of  a  few  manuals;  other 
instsnces—those  that  have  veritable  libraries  of  documentation — might 
require  a  documentation  computer  system  of  significant  size.  In  an 
experimental  system  on  an  aircraft  carrier,  for  example,  the  computer 
system  that  handles  documentation  is  a  network  of  about  30  PERQs*  that 
are  16-bit,  chip-based  "personal"  computers  of  substantial  capability. 


Documentation  as  Part  of  an  Overall  System 

The  aircraft  carrier  project  Introduces  a  concept  that  will  no  doubt  be 
very  important  in  the  future:  Documentation  and  what  users  do  with  it 
are  parts  of  a  larger  system.  If  the  use  of  documentation  leads  to  the 
discovery  of  a  defective  part.  Inventory  must  be  checked  and  ordering  may 
have  to  be  done.  If  the  use  of  documentation  leads  to  isolation  of  a 
software  bug,  software  maintenance  work  must  be  done.  It  would  be 
convenient  and  would  foster  efficiency  if  the  same  system  that  handled 
documentation  also  handled  Inventory  and  software  maintenance.  To 


*PERQ  is  a  trademark  of  the  Three  Rivers  Computer  Corporation. 
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improve  the  overall  effectiveness  of  documentation,  research  is  needed  on 
the  Interactions  of  documentation  with  other  parts  of  the  overall  task 
support  system. 

* 

Computer-Based  Versus  Print-on-Paper  Documentation 

The  discussion  thus  far  has  focused  on  computer-based  documentation,  even 
when  the  system  being  documented  is  not  Itself  an  Interactive  computer 
system.  That  choice  reflects  the  judgment  that  research  in 
computer-based  documentation  is  more  likely  to  make  a  major  payoff  than 
ongoing  research  in  prlnt-on-paper  documentation.  The  latter  research 
has  led  to  many  improvements  and  the  total  effect  has  been  significant, 
but,  insofar  as  conventional  documentation  is  concerned,  diminishing 
returns  have  set  in.  Computer-based  documentation,  by  contrast,  with  the 
capability  of  the  computer,  offers  hope  of  a  very  major  advance.  While 
computer-based  documentation  is  not  a  new  concept  by  any  means,  it  has 
just  recently  begun  to  be  studied  systematically.  The  "help  systems”  and 
the  "tutorials"  of  the  1960s  and  1970s  were  written  without  the  benefit 
of  research  of  the  kind  that  was  devoted,  for  example,  to  programming 
languages.  As  a  result,  it  has  been  said,  the  help  systems  needed  help 
systems  and  the  tutorials  needed  tutors.  Our  conclusion  Is  that  now  Is 
the  time  to  make  a  strong  research  attack  on  computer-based 
documentation,  Including  self-instructional  programs,  coherent 
8)'6tem-wlde  help  systems,  documentation  keyed  to  the  behavior  of  programs 
(so  that  an  error  calls  forth  an  explanation  of  what  went  wrong),  and 
programming  languages  that  write  programs  to  explain  themselves. 


Capturing  the  Intent  of  the  Creators  cf  the  System 


As  suggested  earlier,  documentation  must  be  viewed  as  a  part  of  the 
overall  system  that  interacts  with  other  parts  of  the  overall  system* 

The  time  dimension— the  history-- -of  the  overall  system  is  a  very 
important  base  of  the  interaction*  Host  systems  are  developed  through 
efforts  to  improve  earlier  systems,  and  those  that  do  not  are  developed 
from  some  kind  of  design  activity  in  the  minds  of  system  designers. 
(Programs  are  systems,  of  couraa,  so  the  same  can  be  said  of  programs)* 
The  intentions  of  the  improvers  and  dtslgners  are  crucially  Important  to 
understanding  what  the  systems  do,  how  they  work,  and  how  they  should  be 
used — but  intentions  tend  not  to  be  captured  In  the  plana  end  designs.  A 
computer  program,  for  example,  usually  tells  how  to  do  something,  not 
what  it  la  that  is  being  done,  and  it  is  vary  difficult  to  reconstruct 
the  programmer's  intentions  from  the  program.  Research  on  this  topic  may 
or  may  not  improve  the  situation,  but  it  clear  that  the  situation  needs 
to  be  Improved.  A  broad  view  of  documentation  ia  Important.  The  right 
approach  may  be  to  create  computer-based  design  and  upgrading 
metasystems,  within  which  improvers  and  designers  would  work  under 
constant  monitoring,  with  as  much  emphasis  on  recording  intentions  and 
goals  as  on  devising  the  means  for  achieving  them.  Note  that  this 
notion,  if  not  developed  with  sensitivity  to  privacy  Issues,  could  lead 
to  serious  ethical  problems. 
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Dynamic  Graphics  and  Documentation 

* 

Although  documentation  was,  in  earlier  days,  primarily  print  on  paper, 
•ome  documentation  has  been  available  In  other  media,  such  as  recorded 
speech  and  movies.  The  latter  offered,  at  considerable  cost,  the 
advantages  of  kinematic  graphics  and  moving  gray~scale  and  color 
pictures.  The  computer  promises  to  reduce  the  cost  of  preparing 
kinematic  graphics  by  having  a  single,  static  program  create  dynamic 
multidimensional  patterne  that  develop  over  time.  The  video  disk 
promises  to  reduce  the  coet  of  storing  and  playing  back  all  kinds  of 
Information,  especially  pictorial  information.  Together  the  computer  and 
the  video  disk  may  open  up  a  new  era  for  dynamic  graphic  documentation. 

At  present  the  computer  can  select  and  present  In  a  few  milliseconds  any 
one  of  the  approximately  55,000  pictures  on  a  video  disk.  It-.can  run  off 
sequences  of  continuous  frames  as  a  movie  or  skip  around  under  program 
control  and  show  fast  slide  sequences.  What  It  selects  can  be 
conditioned,  of  course,  by  the  *esponses  of  the  viewer  or  viewers.  These 
capabilities  present  an  exciting  opportunity  to  explere  and  develop  new 
approaches  to  documentation. 

Another  exciting  opportunity  Is  being  studied  under  the  rubric  of 
program  visualization.  The  computer  is  capable,  of  course,  of  displaying 
representations  of  Its  own  Internal  operation.  It  can  present  sequences 
of  symbols  representing  the  program  that  Is  being  executed  and  the  date 
on  which  the  program  le  operating.  Alternatively,  It  can  present  graphs, 
diagrams,  and  pictures  to  tell  the  person  at  the  console  what  the  program 
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should  be  doing  and  what  it  ia  in  fact  doing.  This  latter  approach  to 
documentation,  which  requires  sophisticated  graphic  display  not  widely 
available  In  the  past,  is  now  economically  as  well  as  technically 
feasible.  The  hope  is  that  iconic  displays  will  prove  superior  to 
symbolic  displays  In  presenting  the  broad  picture  of  the  behavior  of 
computer  programs  and  systems  and  in  helping  people  deal  with  their 
intrinsic  complexity.  With  the  Iconic  approach,  it  may  be  possible  to 
provide  something  analogous  to  a  zoom  lens,  through  which  one  would  be 
able  to  monitor  and  control  the  broad  picture  as  long  as  everything 
proceeds  according  to  plan,  then  focus  on  the  offending  details  as  soon 
as  trouble  arises. 


Documentation  in  the  Form  of  Knowledge  Bases 

Conventional  documentation  takes  the  forms  of  natural  language  text, 
diagrams,  sketches,  pictures,  snd  tables  of  data;  it  is  designed 
exclusively  to  be  read  by  eye.  New  forms  of  documentation  are  becoming 
essential:  pointer  structures,  semantic  networks,  procedural  networks, 
and  production  rules,  documentation  designed  to  be  interpreted  by 
computer  programs.  Such  documentation  will  probably  be  used  first  in 
interactive  computer  systems  to  help  end  users  or  programmers  and 
maintenance  workers,  but  in  due  course  it  will  be  used  also  in  fully 
automatic  systems  sophisticated  enough  to  read  their  own  documentation 
and  restructure  themselves  to  overcome  difficulties  and  maximize 
performance.  Some  work  has  already  been  done  on  such  documentation  in 
the  field  of  artificial  intelligence;  much  more  needs  to  be  done.  It  is 
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essential  to  couple  research  on  documentation  closely  with  other  research 
pertinent  to  the  systems  In  which  it  will  be  used — for  example,  with  work 
on  Interactive  tutorial  systems  for  end  users,  Interactive  maintenance 
systems,  and  robotic  maufacturlng  systems. 


Computer  Systems  to  Facilitate  Conventional  Documentation 

The  foregoing  emphasis  on  computer-based  documentation  expresses  our 
conviction  that  It  Is  the  high-payoff  area  within  the  documentation 
field,  but  It  should  not  be  taken  to  imply  that  conventional 
documentation  Is  dead.  We  think  that  two  main  foci  have  the  greatest 
potential  payoff  for  research  In  conventional  documentation:  (1) 
understanding  the  target  group  of  people  that  the  documentation  Is 
Intended  to  help  and  the  tasks  In  which  they  will  be  engaged  when  they 
use  the  documentation  and  (2)  using  computer  systems,  with  good  editors, 
formatters,  and  composers  to  facilitate  creation  and  production  of 
conventional  documentation. 

The  theme  of  understanding  the  users  is  developed  elsewhere  in  this 
chapter.  Great  advances  have  been  ma^e  in  the  last  few  years  In  the 
design  of  computer-based  systems  for  creating  and  producing  conventional 
documents,  and  research  In  that  area  has  much  new  technology  to  work  on. 
Indeed,  research  is  needed  to  develop  the  capability  to  make  the  new 
editors,  formatters,  and  composers  easy  to  use  in  order  to  facilitate  the 
preparation  of  documentation  that  will  make  them  and  other  systems  easy 
to  use.  Kruesl,  for  example,  supported  by  the  Office  of  Naval  Research 
(NR  196-160),  is  Investigating  the  relationship  between  the  types  of 
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documentation  provided  to  programmer!  and  their  performance  on  a  wide 
variaty  of  software-related  tasks. 

In  summary,  research  should  be  emphasized  in  several  areas  pertinent 
to  documentation:  (1)  techniques  of  on-line  documentation,  (2) 
interactions  and  information  flows  between  document  subsystems  and  other 
subsystems,  (3)  efforts  to  capture  the  intent  of  designers  and  upgraders 
of  systems,  (4)  dynamic  graphics  and  the  video  disk,  (5)  dynamic  graphics 
and  program  visualization,  (6)  knowledge  bases,  (7)  understanding  the 
uses  and  users  of  documentation,  and  (8)  computer-based  systems  for  the 
development  of  conventional  documentation.  Of  these  suggestions  two 
primary  research  needs  are  to  know  how  and  when  to  use  display 
documentation  with  graphics  and  what  program  visualization  techniques  are 
most  helpful  to  users. 


SUMMARY  AMD  CONCLUSIONS 

The  primary  research  recommendatons  in  the  areas  of  users,  tasks, 
hardware,  software,  and  documentation  Include  a  major  emphasis  on 
developing  new  methodologies  to  evaluate  what  is  meant  by  ease  of  use  in 
human-computer  interaction.  Does  ease  of  use  mean  the  extent  to  which  it 
is  easy  to  learn  to  use  a  computer;  does  it  imply  good  design  of  hardware 
and  software  for  a  variety  of  naive,  casual,  and  professional  users;  does 
it  mean  that  any  task  can  be  done  quickly  and  without  errors;  does  it 
encompass  a  component  of  judged  satisfaction  about  use;  or  does  it  mean 


all  of  these? 


We  n««d  to  know  what  user  characteristics  are  important  determinants 

of  successful  human-computer  interaction  for  a  specified  set  of  tasks, 

such  as  data  base  inquiries,  computation  and  accounting  problems,  and 
* 

editor  or  word  processing  functions.  In  the  area  of  hardware  design, 
more  research  is  needed  to  evaluate  alternatives  to  keyboard  input 
(including  voice  input),  uses  of  color  in  displays,  the  best  slses  of 
displays,  and  alternatives  to  CRT  displays.  Studies  in  evaluating 
software  are  barely  beginning  to  provide  data  for  design  use.  We  don't 
yet  know  how  to  conduct  systematic  research  studies  in  software  design, 
what  Independent  variables  are  most  important,  and  what  dependent 
variables  of  human-computer  interaction  should  be  recorded.  We  don't 
have  data  to  support  the  design  of  a  simulation  facility  to  effectively 
evaluate  commands  in  operating  systems,  editing  systems,  knowledge-based 
systems,  and  query  systems.  We  need  to  understand  users'  conceptual 
models  in  Interacting  with  specific  software  systems,  and  we  need  more 
information  about  the  advantages  and  disadvantages  of  natural. .language 
software  systems.  Documentation  may  veil  become  part  of  the  available 
software  for  users;  when  and  how  to  display  documentation  is  an  important 
area  for  research.  Research  is  needed  on  how  best  to  use  graphics  and 
special  knowledge  bases  to  facilitate  uses  of  documentation  either  on 
line  or  In  manuals.  Current  documentation  is  designer-oriented  rather 
than  user-oriented,  and  the  perspectives  should  be  changed  so  that 
documentation  is  used  more  effectively. 

Although  the  research  needs  outlined  are  numerous,  a  major  emphasis 
in  this  chapter  is  on  systematic  studies  that  Include  all  four 
substantive  variables— user  and  task  characteristics,  hardware,  software, 
and  documentation— and  the  interaction  of  these  components  with  a 
clear-cut  set  of  studies  to  define  ease  of  use. 
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VI 

POPULATION  GROUP  DIFFERENCES 


Many  areas  of  research  in  human  factors  have  concentrated  on  systems  that 
fit  the  average  person.  In  those  studies,  Individual  differences 
traditionally  have  been  treated  as  little  more  than  an  error  problem. 

Thus  few  data  are  available  in  many  areas  of  human  factors  on  the 
Interaction  of  different  systems  with  variables  such  as  ability  levels  or 
age  levels.  Attempts  to  classify,  describe,  predict,  and  exploit 
Individual  and  group  differences  extend  to  the  beginnings  of  recorded 
history.  Some  of  the  earliest  decipherable  samples  of  writings  Include 
references  to  the  physical  and  mental  differences  between  men  and  women, 
serfs  and  noblemen,  slaves  and  masters,  and  barbarians  and  civilized 
persons.  It  was  not  until  the  nineteenth  century,  however,  that  the 
study  of  individual  and  group  differences  assumed  the  systematic  and 
rigorous  qualities  of  scientific  investigation.  Ihe  attempts  of  Sir 
Francis  Galton  (1822-1911)  to  describe  the  nature  of  individual 
differences  are  the  foundations  of  what  is  sometimes  referred  to  as 
differential  psychology. 


The  principal  authors  of  this  chapter  are  Irwin  L.  Goldstein  and  Alphonse 
Chapanls . 
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Since  Galton,  investigations  of  individual  and  group  differences 
carried  out  by  psychologists,  anthropologists,  and  sociologists  number  in 
the  hundreds  of  thousands.  There  is  a  psychological  journal.  The  Journal 
of  Cross-Cultural  Psychology,  entirely  devoted  to  studies  of  this  kind. 
One  of  the  most  important  applications  of  this  work  in  psychology  has 
been  the  development  of  a  multimillion  dollar  testing  industry. 
Psychologists  have  devised  hundreds  of  tests  of  ability,  achievement, 
skills,  knowledge,  and  personality  (Buros,  1978)  that  are  used  routinely 
for  classifying  and  selecting  employees  for  thousands  of  jobs  and 
occupations.1 

One  of  the  most  ambitious  and  thorough  attempts  to  relate  individual 
characteristics  of  workers  to  job  requirements  is  the  Dictionary  of 
Occupational  Titles  (U.S.  Department  of  Labor,  1977).  This  compendium 
gives  profiles  of  the  educational,  aptitude,  Interest,  physical,  and 
temperament  characteristics  required  of  a  worker  to  achieve  average 
successful  job  performance  in  thousands  of  occupations.  The  military 
services  have  tried  to  do  something  similar  on  a  more  modest  scale.  In 
the  preparation  of  personnel  requirements  data,  the  Air  Force  Design 
Handbook  (Air  Force  Systems  Command,  1969)  specifies  that  tasks  should  be 
rated  along  six  dimensions;  ambient  environment,  equipment 
characteristics,  mental  demands,  physical  demands,  hazard  exposure,  and 
task  criticality.  Figure  6-1  shows  the  three  levels  of  mental  demands 
that  may  be  required  of  people  by  various  duties  and  tasks.  - 


* Tests  are  also  used  for  other  purposes,  for  example,  diagnosing  and 
classifying  mental  illnesses,  but  our  concern  here  is  with  job-related 
activities . 
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CODE  1  requires  little  or  no  formal  training,  just  a  basic  Introduction 
to  the  task;  ability  to  follow  relatively  simple  written  or  oral 
Instructions;  little  judgment,  since  only  elementary  decisions  Involved; 
little  concentration;  little  or  no  recall  of  relevant  knowledge  for 
decisions  or  Inference;  only  precise  determinations,  such  as  GO/NO-GO, 
UP/DOWN,  MORE/LESS,  YES/NO,  ALL/NONE,  CORRECT/ INCORRECT,  etc. 


CODE  2  requires  moderate  technical  knowledge  and  training;  6ome  ability 
to  adjust  to  changing  situations;  occasional  exercise  of  judgment 
Involving  use  of  technical  knowledge;  ability  to  understand  and  use 
technical  manuals;  some  Initiative  and  Ingenuity  required;  occasional 
recall  of  relevant  knowledge  and  experience  of  the  practical  type  for 
decisions  or  Inferences;  decisions  involving  somewhat  detailed  procedures 
or  measurements,  as  In  assembling,  disassembling,  Installing,  removing, 
Inspecting,  testing,  operating,  adjusting,  computing,  monitoring, 
servicing,  etc. 


CODE  3  requires  a  high  degree  of  complex  and  varied  technical  knowledge, 
with  considerable  formal  and  Informal  training;  a  high  degree  of 
continuous  concentration,  with  attention  to  advanced  and  involved 
elements  of  the  task;  continuous  exercise  of  a  high  degree  of  judgment, 
with  decisions  based  on  varied  and  complex  factors  requiring 
understanding  of  underlying  principles  and  procedures;  extensive  recall 
of  relevant  and  precise  knowledge  and  experience  for  decisions  and 
inferences;  frequent  decisions  at  the  theoretical  and  abstract  level; 
precise  and  detailed  analysis,  correlating,  computing,  organizing,  and 
sequencing  of  processes  or  data,  as  In  variable  emergency  procedures, 
troubleshooting,  planning,  scheduling,  etc. 


FIGURE  6-1  Classification  of  the  Mental  Demands  Made  on  Fersonnel  by 
Duties  and  Tasks 

Source:  Air  Force  Systems  Command  (1969). 
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Although  it  Is  seldom  explicitly  stated,  the  underlying  rationale  of 
most  of  these  classifications  is  that  the  job  or  the  occupation  is  a 
given,  a  fixed  quantity.  The  aim  of  personnel  selection  is  therefore  to 
find  persons  who  have  the  abilities,  skills,  and  other  characteristics 
required  to  perform  particular  jobs.  From  the  standpoint  of  human 
factors,  however,  a  job  is  not  a  fixed  quantity  but  rather  something  that 
can  be  modified  and  designed  to  fit  people  with  varying  characteristics. 
Thus  it  becomes  important  to  know  in  what  ways  people  vary  and  by  how 
much.  In  this  area  there  are  serious  gape  in  our  knowledge.  The  most 
thorough  translation  of  individual  difference  data  into  design 
requirements  has  been  done  in  the  field  of  anthropometry,  which  involves 
measurement  of  the  human  body.  It  is  possible  to  write  equipment  design 
specifications  so  that  the  equipment  will  fit  90  percent,  95  percent,  or 
any  other  proportion  of  a  particular  user  population.  The  information 
necessary  to  write  equally  precise  design  specifications  for  other  human 
dimensions  and  characteristics,  however,  is  not  available. 

Attempts  have  been  made  to  do  that,  but  further  research  is  needed  on 
this  complex  problem.  The  Air  Force’s  six  task  dimensions  of  ambient 
environment,  equipment  characteristics,  physical  demands,  hazard 
exposure,  and  task  criticality  are  a  good  initial  effort  (see  Table  6-1), 
yet  the  Air  Force  Design  Handbook  acknowledges  its  limitations:  "Because 
of  the  broad  range  of  equipment  characteristics,  complete  criteria  are 
not  presented  here.  The  following  are  merely  suggested  guidelines” 
(Section  DN4C3,  p.  13).  For  example,  the  manual  states  that  Code  1 
equipment  is  .  .  complex  but  adequately  designed  for  ease  of  use.  .  . 

What  the  definition  does  not  specify  is  ease  of  use  for  whom. 
Something  that  is  easy  for  an  astronaut  to  use  may  be  completely  beyond 


s 
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the  capabilities  of  an  individual  with  only  an  elementary  school 
education.  To  state  the  problem  explicitly,  we  do  not  know  exactly  how 
to  design  complex  equipment  so  that  it  can  be  used  with  ease  by  people 
with  average  IQs,  people  with  IQs  as  low  as  80,  people  with  fifth-grade 
reading  abilities,  or  people  for  whom  English  is  a  second  language. 


THE  IMPACT  OF  FEDERAL  ANTIDISCRIMINATION  LEGISLATION 


Antidiscrimination  legislation  has  focused  attention  on  human  factors 
issues  related  both  to  complying  with  legislative  requirements  and 
maintaining  the  productivity  of  a  work  force  with  greater  diversity  than 
in  the  past.  As  a  result  there  is  Increased  concern  over  the  Interaction 
of  individual  differences  with  programs  such  as  job  redesign  and  training 
as  well  as  over  organizational  attitudes  toward  various  populations 
(e.g.,  the  elderly)  that,  may  constrain  their  performance. 

As  a  result  of  the  U.S.  Civil  Rights  Act,  federal  guidelines  have 
been  developed  concerning  personnel  decisions  that  affect  protected 
classes,  which  include:*  American  Indian  or  Alaskian  natives,  blacks  not 
of  Hispanic  origin,  Hlspanlcs,  and  Asian  or  Pacific  Islanders.  In 
addition,  federal  legislation  has  made  it  illegal  to  discriminate  on  the 
basis  of  sex,  age,  or  disability.  Any  personnel  action  resulting  in 
adverse  impact  against  any  of  these  groups  can  result  in  litigation.  In 
this  context,  personnel  decisions  are  not  limited  to  selection  or 
promotion  but  rather  refer  to  any  personnel  practice,  such  as  job  and 
workplace  redesign,  selection  for  training,  and  the  use  of  training  as  a 
basis  for  promotion. 


Legal  actions  resulting  from  charges  of  discrimination  have 

stimulated  research  on  the  procedures  necessary  to  assess  the  validity  of 
•> 

these  types  of  personnel  practices;  however,  most  of  the  emphasis  has 
been  on  the  establishment  of  procedures  to  validate  selection  tests 
(American  Psychological  Association,  1980).  Similar  concerns  are  being 
expressed  about  methodologies  for  evaluating  training  and  job  redesign 
(Bartlett,  1978).  The  research  emphasis  has  been  on  establishing  data 
bases,  so  that  It  is  possible  to  design  programs  that  do  not  have  adverse 
impact. 

As  a  consequence  of  antidiscrimination  legislation  as  well  as  social 
and  economic  factors,  people  from  special  population  groups  are  moving 
into  occupations  that  were  previously  considered  nontradltional  for 
them.  An  example  is  women  who  are  entering  managerial  and  blue-collar 
jobs  and  the  military  services.  The  military  services  are  also  accepting 
more  people  (male  and  female)  who  have  lower  ability  as  measured  by 
traditional  academic  aptitude  measures.  These  changes  in  the  composition 
of  the  work  force  and  the  armed  services  have  revealed  an  Important 
problem  in  addition  to  the  human  factors  issues  of  designing  jobs, 
equipment,  and  training  to  accommodate  individual  differences:  It  has 
only  recently  been  recognized  that  organizational  attitudes  toward  people 
entering  nontradltional  jobs  may  adversely  affect  productivity  by 
hindering  their  performance  and  constraining  occupational  aspirations. 
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SEX  AND  JOB  PERFORMANCE 

Sheridan '8  (1975)  description  of  the  American  Telephone  and  Telegraph 
Company's  experience  In  placing  women  in  craft  jobs  illustrates  the 
Implications  of  human  factors  for  sex  and  job  performance.  Despite 
rigorous  recruiting  and  comprehensive  training  efforts,  the  women 
recruited  into  a  particular  job  dropped  from  training  at  an  average  rate 
of  50  percent,  and  the  women  who  completed  training  usually  did  not  last 
a  full  year  on  the  job.  A  task  analysis  of  the  job  indicated  that  the 
physical  tasks  were  extremely  difficult  for  women  to  perform; 
furthermore,  this  analysis  determined  which  tasks  were  causing  the  most 
difficulty.  Some  of  the  most  serious  problems  centered  on  the  use  of  a 
ladder  that  weighed  approximately  80  lbs.  and  was  14  feet  long  before 
being  extended.  Women  had  great  difficulty  placing  the  ladder  against  a 
building  because  they  had  to  apply  force  below  the  midpoint  of  the  ladder 
just  as  the  force  required  to  raise  it  was  increasing.  A  fiberglass  tube 
was  connected  to  the  top  rungs  of  the  ladder  that  enabled  the  worker  to 
push  the  ladder  against  the  building  much  more  easily.  As  a  result, 
workers  who  were  5  foot  2  Inches  weighing  120  pounds  were  able  to  raise  a 
72  lb.  ladder  with  one  hand.  These  and  other  design  modifications  not 
only  allowed  women  to  perform  the  job  but  also  resulted  in  fewer  back 
injuries  for  men. 


AGE  AND  JOB  PERFORMANCE 


Important  considerations  with  regard  to  age  and  job  performance  are  that 
the  average  age  of  the  population  is  increasing  and  both  age 
discrimination  legislation  and  rulings  against  forced  retirement  are 
resulting  in  a  larger  number  of  older  people  in  the  work  force.  Many  of 
these  individuals  will  require  additional  training  as  a  result  of  job 
shifts,  technological  changes,  or  simply  Interest  in  a  new  career.  The 
biases  operating  against  these  people  are  made  obvious  by  Britton  and 
Thomas’s  (1973)  study  of  the  views  of  employment  Interviewers.  They 
noted  that  50-year-old  workers  were  viewed  as  the  most  difficult  to  place 
during  a  recession,  the  most  difficult  for  an  employer  to  train,  and  the 
least  able  to  maintain  production  schedules.  These  views  are  based  on 
preconceived  beliefs  that  older  workers  cannot  perform  as  well  on  the  job 
and  cannot  easily  acquire  new  skills.  Data  relevant  to  these  questions 
are  virtually  nonexistent;  a  thorough  review  (Fozard  and  Popkin,  1978)  of 
perceptual  and  cognitive  data  analyzed  by  age  reinforces  the  view  that 
there  are  few  data  relevant  to  work  situations.  Much  of  that  review  is 
based  on  data  from  laboratory  experiments  on  topics  such  as  paired 
associate  learning,  iconic  memory,  and  visual  discrimination,  making 
generalizations  to  work  situations  hazardous  at  best. 

The  deficient  6tate  of  this  research  is  summarized  in  Sheppard's 
(1970)  generalizations  about  basic  research  on  aging  end  job 
performance:  The  research  fails  to  differentiate  various  aspects  of  the 
work  situation,  Including  physical,  psychomotor,  sensory,  and  social 


10 


characteristics;  most  of  the  emphasis  is  on  average  performance,  with 
little,  if  any,  attention  to  the  substantial  number  of  individual 
differences;  and,  there  is  a  blind  faith  in  trend  extrapolations.  If 
workers  ages  30-40  have  lower  morale  than  workers  ages  20-30,  it  is 
simply  assumed  that  workers  ages  40-50  will  have  even  lover  morale. 

A  good  example  of  the  implications  of  our  lack  of  knowledge  is 
evidenced  by  the  continuing  controversy  concerning  airline  pilot  age, 
health,  and  performance.  An  Institute  of  Medicine  (1981)  report  notes 
that  although  the  average  risk  of  acute  Incapacitation  Increases  with 
age,  there  are  large  individual  differences.  In  addition,  while  there 
are  decreases  in  capacity,  speed  or  accuracy  of  attention,  memory,  and 
Intellectual  skills  with  Increasing  age,  there  is  also  evidence  that 
well-practiced  skills  may  not  show  any  age-related  decline.  The  report 
concludes  that  there  is  a  need  for  research  on  age-related  changes  among 
pilots  and  a  need  for  research  on  pilot  performance  on  tasks  that  are 
representative  of  actual,  work  situations. 

Of  more  immediate  relevance  to  this  report  are  the  relationships 
between  group  variables  such  as  age  and  equipment  design.  For  example, 
as  they  age,  many  people  require  the  use  of  bifocals.  How  does  the  use 
of  bifocals  relate  to  the  need  to  read  information  from  displays  such  as 
those  found  on  word  processing  equipment?  Is  it  possible  that  the 
displays  must  be  designed  differently  or  that  the  information  must  be 
displayed  differently  depending  on  the  age  of  the  operator?  Questions 
such  as  these  constitute  a  largely  unexplored  topic  for  research. 


INTERACTIONS  AMONG  VARIABLES 


Another  serious  gap  In  our  knowledge  is  how  various  Individual  and  group 
differences  interact  to  affect  job  performance.  For  example,  there  are 
considerable  data  available  relating  aging  to  maximum  oxygen  uptake, 
which  determines  the  capacity  of  an  individual  to  do  prolonged  heavy  work 
(Astrand  and  Rodahl,  1977).  These  data  show  that  there  is  a  steady 
decrement  In  aerobic  power  beginning  at  about  age  20,  such  that  a 
60-year-old  attains  about  70  percent  of  the  maximum  of  a  25-year-old. 
Unfortunately,  there  are  a  few  data  on  most  population  differences  or 
Individual  differences  as  they  are  related  to  work  situations.  McFarland 
and  O'Doherty  (1959)  concluded  the  following  regarding  the  relationship 
of  aging  and  work  performance  (pp.  454-455): 

Although  most  studies  show  an  unrelieved  picture  of  decline  in 
capacities,  it  Is  well  to  remember  that  this  constantly  changing 
balance  between  physiological  and  psychological  Impairment,  on  the 
one  hand,  and  increased  experience,  wisdom,  and  judgment,  on  the 
other,  occasionally  results  In  actual  Improvement  of  capacities, 
especially  In  those  functions  which  are  of  greatest  importance  in 
dally  living. 

These  and  other  interactions  of  variables  are  another  almost  completely 
untapped  area  of  research. 
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NATIONAL  AND  ETHNIC  DIFFERENCES 

There  are,  of  course,  other  important  differences  in  population 
characteristics  that  should  be  considered  in  Job  redesign  and  training 
systems.  National  and  ethnic  differences  have  implications  for  equipment 
design  that  have  just  recently  begun  to  be  Investigated  (Chapanls, 

1975).  These  differences  are  reflected  in  anthropometric,  physiological, 
psychological,  language,  and  cultural  variables  that  affect  equipment 
design. 

For  example,  Ruf fell- Smith  (1975)  notes  that  telegraph  systems  were 
originally  used  as  communication  devices  in  air  traffic  control  systems; 
however,  with  the  Increased  amount  of  speed  of  air  traffic,  voice 
communication  systems  replaced  telegraph  devices.  Obviously,  the  use  of 
the  different  languages  of  the  many  nations  involved  In  air  travel  was  a 
serious  Impediment  to  the  operation  of  voice  systems.  After  World  War  II 
English  was  chosen  as  the  language  of  use  because  at  that  time  most 
aircraft  were  operated  by  English-speaking  countries.  Yet  there  is  a 
vide  variation  in  English  dialects  and  pronunciation,  to  the  extent  that 
some  dialects,  such  as  that  spoken  in  Newcastle,  are  not  understood  by 
people  elsewhere  in  the  British  Isles.  Obviously  the  problems  are  more  _ 
severe  when  the  speaker's  native  language  is  not  English. 

Ruffell-Smith's  analysis  of  communication  errors  indicates  that  this 
problem  can  be  serious  in  air  traffic  communication,  especially  when  the 
speed  of  reaction  is  a  critical  element  in  avoiding  an  accident. 

Clearly,  the  Implications  of  these  population  differences  should  be 
considered  in  design  decisions. 
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Ethnic  Variables  in  Hunan  Factors  Engineering  (Chapanis,  1973) 
provides  other  examples  of  equipment  design  complexities  caused  by 
language  differences.  One  chapter  (Hanes,  1975)  shows  the  variety  of 
accounting  keyboards  that  have  been  designed  to  accommodate  Borne  of  the 
European  and  Mideast  languages.  Another  chapter  (Brown,  1975) 
illustrates  the  design  problems  that  were  encountered  in  designing  a 
computer  terminal  for  Japanese,  a  language  that  is  markedly  different 
from  the  Indo-European  languages.  In  general,  there  is  little 
appreciation  of  the  problems  involved  in  designing  equipment  for  diverse 
national  and  ethnic  groups.  The  Human  Engineering  Guide  to  Equipment 
Design  (Van  Cott  and  Klnksde,  1972)  is  the  best  single  source  of  human 
factors  data  available,  yet  it  is  almost  entirely  concerned  with  American 
and  European  data.  It  is  necessary  to  learn  to  what  extent  its  data  and 
design  recommendations  need  to  be  modified  or  supplemented  for 
International  use. 


INDIVIDUAL  DIFFERENCES  AND  TRAINING 


Closely  related  to  problems  of  equipment  design  are  those  associated  with 
the  training  of  individuals  to  operate  complex  equipment.  Here  again  our 
information  is  seriously  deficient.  An  approach  that  has  some  promise  16 
the  aptitude-treatment  interaction  (ATI)  model.  The  goal  of  this 
approach  is  to  match  a  particular  mode  of  instruction  to  an  individual '6 
distinctive  characteristics  so  that  each  person  is  assigned  the  most 
appropriate  learning  procedure.  A  dlsordinal  aptitude-treatment 
interaction  is  one  in  which  individuals  with  high  aptitude  perform  besi. 
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with  one  treatment  (e.g.,  training  or  display),  while  those  with  lower 
aptitude  perform  best  with  another  treatment.  Thus,  the  aptitude  level 
of  the  individual  determines  the  form  of  treatment  that  has  the  best 

l 

chance  of  success.  Aptitude  in  this  context  refers  to  any  personal 
characteristics  that  relate  to  learning  and  so  can  include  a  broad  range 
of  variables,  such  as  styles  of  thought,  personality,  and  various 
scholastic  aptitudes.  Treatment  has  typically  referred  to  instructional 
inodes  like  programmed  instruction,  computer-assisted  instruction,  visual 
versus  verbal  presentations,  etc;  it  can  be  generalized,  however,  to  any 
Intervention,  including  job  redesign. 

An  exhaustlve"sreyiew  of  this  appealing  strategy  is  provided  in  the 
text  by  Cronbach  and  Snow  (1977).  They  examined  a  large  number  of 
potential  aptitudes,  such  as  learning  rates,  abilities,  and  personality, 
and  considered  their  interactions  with  various  instructional  techniques. 
While  early  reviews  of  this  topic  were  more  pessimistic,  Cronbach  and 
Snow's  extensive  review  and  reanalyses  of  data  have  led  them  to  conclude 
that  aptitude  treatment  interaction  effects  are  real  phenomena.  They 
note  that  the  findings  that  most  clearly  suggest  ATI  effects  are  those 
dependent  on  prior  learning  experience:  The  technique  that  works  best  Is 
the  one  that  an  individual  has  already  experienced.  However,  ATI  effects 
have  not  often  been  generalized  or  replicated.  Goldstein  (1980)  notes 
the  need  for  systematic  empirical  and  theoretical  research  that  matches 
Individual  differences  among  learners  to  various  instructional 
strategies.  The  haphazard  assignment  of  individuals  with  particular 
abilities  to  any  available  Instructional  technique  is  not  likely  to 


produce  dividends 
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BARRIERS  TO  SUCCESSFUL  PERFORMANCE 

Another  Important  topic  Is  the  identification  of  barriers  to  successful 
performance  for  different  groups.  For  example,  some  employment 
interviewers  perceive  women  as  more  likely  to  be  absent  and  to  have  fewer 
skills,  even  though  they  have  no  evidence  to  support  these  beliefs 
(Britton  snd  Thomas,  1973).  Similarly,  the  elderly  are  viewed  as 
difficult  to  train  (Britton  and  Thomas,  1973).  Researchers  concerned 
with  these  Issues  emphasise  that  the  identification  of  organizational 
constraints,  in  military  organizations  for  example,  is  a  first  step  in 
understanding  and  resolving  their  serious  retention  problem.  One  study 
(Boyd  et  al.,  1975)  of  1,573  women  in  their  first  tour  in  the  Army'B 
basic  training  program  was  critical  of  the  program's  failure  to  provide 
realistic  expectations  about  the  training  process.  Subsequent  to  the 
basic  training  program  supervisors  reported  the  main  difference  between 
good  and  poor  performers  was  job-related  attitudes  (discipline,  following 
orders,  military  courtesy)  that  were  not  adequately  presented  in  basic 
training. 


RECOMMENDATIONS  FOR  RESEARCH  ON  POPULATION  GROUP  DIFFERENCES 

A  research  program  to  explore  issues  concerning  population  group  8nd 
individual  differences  would  need  to  take  several  approaches: 

(1)  It  is  necessary  to  conduct  literature  reviews  and  examinations 
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of  reports  that  forecast  which  type  of  population  group  variables 
(such  as  age  and  sex)  and  which  type  of  work  situation  parameters 
(such  as  visual  displays  on  a  word  processor)  will  be  important  in 
the  future. 

(2)  It  is  necessary  to  collect  and  examine  available  theories  and 
empirical  data  about  the  relevant  parameters  (e.g.,  changes  in 
information  processing  capability  as  a  function  of  age). 

(3)  Research  should  be  sponsored  on  a  number  of  topics: 

o  The  relationship  between  population  group  variables  and 
performance  on  relevant  work  tasks, 
o  The  interaction  between  population  group  differences  and 
various  interventions,  such  as  job  redesign  and  training, 
o  The  specification  of  design  changes  based  on  research 
findings  resulting  from  these  research  recommendations. 

(4)  In  addition,  data  should  be  collected  and  analyzed  to  identify 
and  remove  organizational  constraints  that  serve  as  barriers  to  the 
successful  performance  of  various  population  groups,  such  as  women 
and  aged  and  handicapped  people. 
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VII 

APPLIED  METHODS  IN  HUMAN  FACTORS 


As  part  of  an  engineering  team,  human  factors,  specialists  apply  their 
knowledge  and  skills  to  system  definition,  design,  development,  and 
evaluation  In  order  to  optimise  the  capabilities  and  performance  of 
human-machine  combinations.  Their  task  can  be  formidable  in  complex 
system  development.  For  example,  military  standard  MIL-H-46855B  of  the 
Department  of  Defense  details  the  human  factors  requirements  that  must  be 
addressed  In  the  development  of  military  systems;  an  outline  of  these 
requirements  appears  as  Figure  7-1.  The  outline  Is  also  a  reasonable 
representation  of  the  human  factors  considerations  that  may  be  relevant 
to  the  development  of  any  system. 

In  designing  and  creating  systems  human  factors  specialists  use  a 
variety  of  analytic  and  data-gatherlng  techniques  to  assess  problems, 
develop  machine  and  human  requirements  and  functions,  and  evaluate  system 


The  principal  authors  of  this  chapter  are  Alphonse  Chapanls  and  Robert  T. 
Hennessy.  It  is  based  on  a  workshop  on  applied  methods  held  in  December 
1981  under  the  sponsorship  of  the  Committee  on  Human  Factors.  The 
workshop  participants  and,  therefore,  the  principal  contributors  to  thi6 
chapter  are  Alphonse  Chapanls  (workshop  chairman),  Johns  Hopkins 
University;  Stuart  K.  Card,  Xerox  Palo  Alto  Research  Center;  David 
Melster,  US  Navy  Personnel  Research  and  Development  Center;  Donald  L. 
Parks,  Boeing  Aerospace  Company;  Richard  W.  Pew,  Bolt  Beranek  &  Newman 
Inc.;  Erich  P.  Prien,  Memphis  State  University;  John  B.  Shafer,  IBM 
Corporation;  and  Robert  T.  Hennessy,  National  Research  Council. 
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3.1 


3.2 


General  Requirements 

3.1.1  Scope  and  Nature  of  Work 

o  Analysis 

o  Design/Development 

o  Test  and  Evaluation 

3.1.2  Human  Engineering  Program  Plan  and  Other  Data 

3.1 .2.1  Human  Engineering  Program  Plan 

3. 1.2. 2  Changes  to  the  Human  Engineering  Program  Plan 
3. 1*2. 3  Other  Data 

3.1.3  Nou  Duplication  (of  Effort) 

Detail  Requirements 

3.2.1  Analysis 

3. 2. 1.1  Defining  and  Allocating  System  Functions 

3. 2. 1.1.1  Information  Flow  and  Processing  Analysis 

3. 2. 1.1. 2  Estimates  of  Potential  Operator/Maintalner 
Processing  Capabilities 

3. 2. 1.1. 3  Allocation  of  Functions 

3. 2. 1.2  Equipment  Identification 

3. 2. 1.3  Analysis  of  Tasks 

3. 2. 1.3.1  Gross  Analysis  of  Tasks 

1.  Determine  System  Performance  Can  Be  Provided  by  Proposed 
Personnel-Equipment  Capabilities 

2.  Assure  Human  Performance  Requirements  Do  Not  Exceed  Human 
Capabilities 

3.  Input  Data  for 

o  Preliminary  Hanning  Levels 
o  Equipment  Procedures 
o  Skill/Training  Requirements 
o  Communication  Requirements 

4.  Critical  Human  Performance 

5.  Possible  Unsafe  Practice 

6.  Promising  Improvements  in  Operating  Efficiency 

3. 2. 1.3. 2  Analysis  of  Critical  Tasks 
1.  Identifying 

o  Information  Required  by  Man,  Including  Task  Initiation 
Cues 

o  Information  Available  to  Man 
o  Evaluation  Process 
o  Decision  Reached  After  Evaluation 
o  Action  Taken 

o  Body  Movements  Required  by  Action 
o  Workspace  Envelope  Required  by  Action 
o  Workspace  Available 

o  Location/ Condition  of  Work  Environment 
o  Frequency/Tolerances  for  Action 
o  Time  Base 

o  Feedback  on  Action  Adequacy 
o  Tools  and  Equipment  Required 


FIGURE  7-1  Outline  of  Human  Factors  Requirements  in  the  Development  of 
Military  Systems 


o  Number  of  Personnel  Required  and  Specialties/Experience 
o  Job  Alds/Reference6  Required 
o  Special  Hazards  Involved 

o  Operation  Interaction  Where  More  Than  One  Crewman  is 
Involved 

o  Operational  Limits  of  Man  (Performance) 
o  Operational  Limits  of  Machine  (State-of-the-Art) 

2.  Covering  All  Affected  Mission/Phases,  Including  Degraded 
Modes  of  Operation 

3. 2. 1.3. 3  Loading  Analysis 

1.  Individual  Crew  Member  Workload  Analysis  Compared  with 

Performance  Criteria 

2.  Crew  Workload  Analysis  Compared  with  Performance  Criteria 
3. 2. 1.4  Preliminary  System  and  Subsystem  Design 

3.2.2  Human  Engineering  Studies,  Experiments  and  Laboratory  Tests 

3. 2.2.1  Studies,  Experiments  and  Laboratory  Tests 

3. 2. 2. 1.1  Mockups  and  Models 

3. 2. 2. 1.2  Dynamic  Simulation 

3. 2. 2. 2  Equipment  Detail  Design  Drawings 

3. 2. 2. 3  Work  Environment,  Crew  Stations  and  Facilities  Design 
o  Atmospheric  Conditions 

o  Weather  and  Climate 
o  Range  of  Accelerative  Forces 
o  Acoustic  Noise,  Vibration  and  Impact  Forces 
o  Provision  for  Human  Performance  During  Weightlessness 
o  Provision  for  Minimizing  Disorientation 
o  Space  for  Crew,  Activity  and  Equipment 

o  Physical,  Visual  and  Auditory  Links  for  All  Man-Equipment 
Interfaces 

o  Safe,  Efficient  Walkways,  Stairways,  Platforms,  Inclines 
o  Provision  to  Minimize  Psychophysiological  Stresses 
o  Provision  to  Minimize  Fatigue — Physical,  Emotional, 

Work-Rest  Cycle 

o  Protection  from  Hazards — Chemical,  Biological, 

Toxicological,  Radiological,  Electrical,  Electromagnetic 
o  Optimum  Illumination  Per  Visual  Tasks 
o  Sustenance  *  Storage  and  Sanitation 

o  Crew  Safety  Protection  Relative  to  Mission  Phase  and  Control- 
Display  Tasks 

3. 2. 2. 4  Human  Engineering  in  Performance  and  Design 
Specifications 

3.2.3  Equipment  Procedure  Development 

3.2.4  Human  Engineering  Test  and  Evaluation 

3. 2. 4.1  Planning 

3. 2. 4. 2  Implementation  (Include  As  Applicable) 

o  Simulation  or  Actual  Conduct  of  Mission/Work  Cycle 
o  Human  Participation  Critical  to  Speed,  Accuracy, 

Reliability,  Cost 


FIGURE  7-1  Continued 
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o  Representative  Sample  of  Non  Critical  Scheduled/Unscheduled 
Maintenance  Tasks 
o  Proposed  Job  Aids 

o  Use  of  Representative  User  Personnel,  Clothing  and  Equipment 
o  Task  Performance  Data  Collection 

o  Task  Performance  Discrepancies — Required  vs.  Obtalaed 
o  Criteria  for  Acceptable  Performance 
3. 2. 4. ,3  Failure  Analysis  (Human  Error  Factors) 

3.2.5  Cognizance  and  Coordination  (Interdisciplinary  Integration) 

3.3  Data  Requirements  Per  Contract  D^ta  List 

3.4  Data  Availability  to  Procuring  Activity 

3.5  Drawing  Approval  by  HFE  for  Man-Machine  Interface 


FIGURE  7-1  Concluded 

Source:  Adapted  from  Parks  and  Springer  (1976) 
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or  subsystem  performance.  Although  many  of  these  problems  would  Ideally 
be  solved  with  the  experimental  methods  used  In  scientific  research, 
practicing  human  factors  specialists  rarely  have  the  luxury  of  using 
properly  counterbalanced  experimental  designs,  with  a  range  of  levels  of 
factors  and  the  precise  control  of  unmanipulated  variables.  This  Is  not 
to  minimise  the  importance  of  experimental  methods  which  are  used 
whenever  possible  and  have  provided  much  of  the  basic  data  in  human 
factors  handbooks*  However,  applied  methods  are  necessary  both  as 
suplements  to  experimental  methods,  e.g.,  for  problem  analysis  and 
structuring,  and  as  substitutes  when  the  pressures  and  constraints  of  the 
engineering  design  environment  preclude  experimental  Investigations. 

Most  practical  work  in  human  factors  is  done  under  conditions  that 
Involve  the  Incomplete  specification  of  system  functions,  complex 
combinations  of  conditions  that  cannot  be  separated  or  controlled, 
restricted  sets  of  alternatives,  limited  time  and  opportunities  for 
Investigation,  and  pressure  to  produce  definitive  results  quickly.  From 
necessity,  human  factors  specialists  have  evolved  an  armamentarium  of 
applied  methods  that  are  appropriate  to  these  conditions  and  that  are 
unfamiliar  to  most  academic  researchers.  These  applied  methods  are 
formal  means  for  acquiring  or  organizing  Information  about  human  factors 
characteristics  that  arise  in  the  context  of  system  design,  development, 
and  evaluation. 

Applied  methods  are  diverse,  reflecting  the  many  purposes  for  which 
human  factors  Information  Is  used.  Some  of  them  come  from  psychology, 
for  example,  questionnaires  and  techniques  for  acquiring,  summarizing, 
and  analyzing  data.  Some  have  been  borrowed,  with  o’-  without 
modification,  from  other  fields,  such  as  industrial  engineering  and  time 
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and  notion  engineering.  For  example,  analytic  methods  draw  heavily  on 
the  engineering  practice  of  systems  analysis,  which  identifies  inputs, 
outputs,  the  functions  performed,  the  range  of  values  that  variables  may 
assume,  process  flow,  the  sequence  of  events,  and  the  timing  of  the 
interrelations  of  system  components.  Other  methods,  such  as  the  critical 
Incident  technique  and  link  analysis,  appear  to  have  been  created  by 
human  factors  specialists  to  meet  their  needs  in  solving  particular 
problems . 

Whatever  their  origins,  applied  methods  have  been  developed  as 
tools  to  help  answer  questions  when  there  are  constraints  of  time, 
dollars,  and  freedom  of  action  and  when  experimental  methods  are  not 
suitable  to  answer  the  questions  that  arise  in  system  development. 
Although  it  is  characteristic  of  applied  methods  that  they  make  it 
possible  to  acquire  and  produce  data  and  information  only  to  the  degree 
of  resolution  and  reliability  sufficient  for  a  particular  purpose,  these 
methods  are  systematic  and  objective  procedures.  That  is,  the  procedures 
are  repeatable  and  input  and  output  data  are  operationally  defined. 

The  importance  of  applied  methods  in  human  factors  work  is  clear 
from  the  number  of  technical  reports  and  journal  articles  that  discuss 
one  or  more  applied  methods.  Two  recent  reports  (Williges  and  Toprailler, 
1980;  Geer,  1981)  list  human  factors  procedures  necessary  for  Air  Force 
system  analysis,  design,  and  evaluation;  the  latter  report  gives  brief 
descriptions  and  critiques  of  approximately  48  human  engineering 
procedures,  the  majority  of  which  are  applied  methods.  Figure  7-2  lists 
applied  methods  that  appeared  in  keyword  lists  of  articles  published 
between  1978  and  1981  in  Human  Factors,  the  journal  of  the  Human  Factors 
Society. 


Accident  studies 
Attitude  studies 
Critical  Incident  studies 
Delphi  techniques 
Fault  tree  analysis 
Functional  analysis 
Lapse  time  photography 
Near-accldent  studies 
Operational  sequence  analysis 
Requirements  analysis 


Activity  analyses 
Cost-benefit  analysis 
Decision  analysis 
Failure  mode  analysis 
Flow  analysis 
Job  analysis 
Link  analysis 
Network  flow  analysis 
Questionnaires 
Task  analysis 


i 


FIGURE  7-2  Applied  Method  Names  Appearing  in  Keyword  Lists  of  Articles 
In  Human  Factors  Between  1976-1981 
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Despite  this  vide  variety  of  applied  methods,  there  is  general 
agreement  among  human  factors  specialists  that  we  need  to  improve 
existing  methods  and  develop  new  ones  (Topmiller,  1981;  Meister,  1982). 
Advances  in  technology,  particularly  In  the  speed,  power,  and  memory  of 
computers,  have  generated  concern  recently  with  the  human  factors 
elements  of  computer  software.  At  the  same  time,  the  explosive  growth  of 
computer  use,  with  resultant  increases  in  the  complexity  and  integration 
of  system  components,  the  automation  of  functions,  and  the  use  of 
artificial  Intelligence,  all  have  profound  methodological  Implications 
for  the  analysis  and  description  of  the  role  of  humans  and  computers  in 
6uch  systems. 

Applied  methods  have  never  previously  been  treated  as  a  single 
topic  deserving  attention  in  its  own  right.*  Consequently,  information 
has  never  been  gathered  on  the  number  and  varieties  of  applied  methods 
available  and  the  frequency  and  adequacy  with  which  they  are  used.  The 
workshop  held  by  the  Committee  on  Human  Factors,  on  which  the  discussion 
in  this  chapter  is  based,  was  an  attempt  by  committee  members  and  a  group 
of  acknowledged  experts  in  applied  methods  to  identify  problems  and  needs 
with  respect  to  applied  methods.  Even  in  the  absence  of  data  on  the 
variety  and  frequency  of  use  of  applied  methods,  we  have  been  able  to 
identify  several  major  problems  and  to  recommend  solutions,  which  may 
make  substantial  improvements  in  practice  possible.  Three  major  problems 
are  discussed;  (1)  the  lack  of  adequate  documentation;  (2)  the  limited 


*Thls  situation  contrasts  with  experimental  methods,  for  which  there  are 
many  textbooks  and  source  books  for  readers  at  all  levels  of 
sophistication. 
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opportunities  available  to  learn  applied  methods,  either  In  colleger  and 
universities  offering  human  factors  courses  or  as  part  of  the  continuing 
education  of  human  factors  specialists;  and  (3)  the  lack  of  research  to 
Improve  existing  methods  and  to  develop  new  methods  that  will  provide  the 
data  and  Information  needed  In  current  and  future  practical  human  factors 
work. 

DOCUMENTATION  OF  APPLIED  METHODS 

The  practical  work  of  human  factors  specialists,  unlike  scientific 
research,  does  not  result  In  an  orderly  progression  and  an  orderly 
accumulation  of  knowledge.  Human  factors  projects  (l.e.,  participation 
In  the  design  of  systems)  and'  the  solution  of  special  problems  come  and 
go  In  great  variety.  Typically  work  Is  performed,  reported,  and 
forgotten  as  new  systems,  and  problems  develop.  Codified,  archival 
repositories  of  practical  work — l.e.,  review  books  and  articles  that 
summarise  the  knowledge  and  procedures  used  in  human  factors  applications 
to  some  point  In  time — are  rare.  As  a  result  the  historical  memory  of 
human  factors  methods  resides  largely  In  the  heads  and  in  the  report 
files  of  practitioners.  By  contrast.  In  the  literature  on  scientific 
research,  the  methods  used  by  investigators  are  maintained  and 
disseminated  in  the  curricula  of  university  departments  and  preserved  on 
library  bookshelves. 

As  an  important  first  step  toward  Improving  knowledge  about  and  use 
of  applied  methods,  we  therefore  recommend  that  one  or  more  projects  be 
Initiated  to  compile  and  review  the  available  information  on  applied 
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methodologies  used  In  human  factor#  and  related  field#,  auch  a# 

Industrial  and  organizational  psychology,  personnel  selection,  and 
Instructional  psychology.  The  object  of  the  review  would  be  to  determine 
what  methods  have  been  used,  how  they  have  been  used,  where  they  are 
used,  and  what  their  advantages  and  disadvantages  are.  The  project 
should  also  include  a  critical  analysis  of  the  methods.  Other  purposes 
of  the  review  would  be  to  structure  or  codify  the  methods  and  to  document 
them  for  subsequent  educational  and  research  purposes. 

It  would  also  be  extremely  valuable  to  practitioners,  educators, 
and  researchers  In  human  factors  to  have  a  compendium  that  codifies  and 
provides  standard  or  generic  descriptions  of  applied  methods  that  are 
used  in  practical  human  factors  work.  Development  of  such  a  compendium 
would  require  a  great  deal  of  judicious  and  careful  effort.  One  of  the 
primary  difficulties  would  be  to  decide  which  methods  are  viable,  valid, 
and  useful.  Because  such  a  compendium  would  necessarily  be  an  implicit 
endorsement  of  the  methods  described,  we  recommend  that  eight  criteria  be 
used  In  the  selection  process.  Methods  that  meet  the  criteria  listed 
below  could  be  regarded  as  having  sufficient  stature  to  be  of  value  in  a 
variety  of  human  factors  applications: 

Importance — Does  the  method  produce  needed  information? 

Cost — Is  the  method  efficient  in  terms  of  effort  and  time? 

Utility-Can  procedures  for  using  the  method  be  easily  interpreted 
and  implemented? 

Available  Input  Parameters — Can  the  necessary  data  be  collected  in 
a  direct,  objective,  and  reliable  way? 
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Usable  Output — Does  the  method  produce  results  that  are 

Interpretable  and  useful  for  decision  making? 

Validity/Verification — Can  or  has  the  method  been  found  to  produce 
the  information  it  is  supposed  to? 

Theoretical  Foundation — Is  the  method  supported  by  accepted 
behavioral  or  measurement  principles? 

Robustness — Can  the  method  be  applied  to  a  variety  of  problems  or 
in  different  contexts? 

These  criteria  Imply  that  the  approach  to  documenting  atandard 
definitions  of  applied  methods  should  be  conservative.  That  is,  only 
those  methods  for  which  there  is  evidence  of  practicality  and  validity 
should  be  selected  for  inclusion  in  a  compendium.  Methods  used  in 
workload  assessment  provide  an  example  of  the  Importance  of  using  these 
criteria.  Measurement  of  workload  is  a  current  topic  of  intense  research 
Interest;  consequently  a  large  number  of  theories,  approaches,  and 
positions  have  been  put  forward.  Since  most  of  the  recent  work  has  not 
been  validated  through  practical  application,  it  would  be  Inappropriate 

I 

to  describe  them  as  standard,  accepted  methods.  Older  methods  exist  for 
assessing  Imposed  workload  that,  while  perhaps  wanting  in  certain 
respects,  have  been  proven  through  repeated  use  to  be  practical, 
reliable,  and  valid  (Parks  and  Springer,  1976)  and  are  likely  to  meet  our 
criteria.  Nevertheless,  there  will  be  hard  choices  to  make  in  deciding 
what  constitutes  an  accepted,  standard  form  of  a  method. 

Multiple  variations  of  a  method  should  probably  not  be  included.  A 
cot&pendium  that  Includes  only  a  set  of  core  methods  that  meet  the 
criteria  would  be  of  great  value  for  both  practical  work  on  system 
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development  and  as  a  foundation  for  the  education  of  human  factors 
students  at  colleges  and  universities.  Attempting  comprehensive  coverage 
of  all  variations  of  methods  would  unnecessarily  complicate  the  task  of 
documentation  and  delay  the  compilation,  causing  confusion  and 
consequently  Inhibiting  its  acceptance.  A  single,  solid  definition  of 
each  particular  method  would  be  most  useful,  since  by  Its  nature  an 
applied  method  undergoes  some  verlation  In  each  instance  of  its  use 
because  of  the  requirements  and  constraints  of  a  particular  project.  In 
the  meantime,  additional  documentation  and  research  to  extend  or  refine 
the  standard  methods  can  be  carried  out. 

In  the  course  of  compiling  a  reasonably  comprehensive  list  of  the 
most  generally  known  applied  methods  (see  Figure  3),  it  became  apparent 
that  the  methodologies  could  be  grouped  into  five  categories  according  to 
their  purpose.  Five  categories  of  applied  methodologies  seem 
appropriate:  analysis.  Identification  of  needs,  date  collection, 
prediction,  and  evaluation.  Each  methodology  appears  only  under  one 
heading,  although  several  of  them  are  appropriate  to  more  than  one 
category. 

The  organization  of  Figure  7-3  is  probably  a  useful  guide  to  the 
scope  of  work  involved  in  documenting  applied  methods.  The  categories 
reflect  a  sequence  of  methods  used,  from  the  early  concept  definition  of 
a  system  to  its  evaluation.  There  is  also  a  rough  correlation  between  - 
the  difficulty  and  detail  involved  in  particular  methods  and  the  stage  of 
application  in  the  process  of  system  development. 

Documentation  of  applied  methods  necessarily  requires  review  of  the 
technical  literature  to  extract  descriptions  of  applied  methods.  To 
expect  a  single  or  a  small  group  of  experts  to  adequately  review  and 
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ANALYSIS 

System  Analysis 
Function/Task  Analysis 
Information  Analysis 
Scenario  Analysis 
Workload  Analysis 
Time-Line  Analysis 
Operational  Sequence  Analysis 
Failure  Mode  Analysis 
Fault  Tree  Analysis 
Link  Analysis 
Function  Allocation 
Anthropometric  Analysis 
Decision  Analysis 
Display  Evaluation  Index 


IDENTIFICATION  OF  NEEDS 

Critical  Incident  Technique 
Surveys/Questionnaires 
Accident  Investigation 
Interviews/Group  Techniques 
Definition  of  User  Population 


DATA  COLLECTION 

Activity  Analysis 
Time  Lapse  Photography 
Real  Time  Film/Video  Recording 
Direct  Observation 
Physiological  Recording 

Quantitative  Performance  Recording  and  Analysis 


PREDICTION 

The  Human  Error  Rate  Procedure  (THERP) 
Data  Store 

Human  Operator  Simulator  (HOS) 

Control  Theory 
Accuracy  Theory 
Predetermined  Time  Analysis 
Readability  Indices 


FIGURE  7-3  Geherally  Known  Applied  Methods  Categorised  by  Purpose 


EVALUATION 


Test  Plan  Evaluation 

Simulation 

Mock-Ups 

Walk  Throughs 

Check  Lists 

Ratings 


FIGURE  7—3  Concluded 


document  the  entire  range  of  applied  methods  would  be  impractical;  a  more 
feasible  approach  would  be  to  subdivide  the  work  according  to  the  five 
categories  of  purpose*  The  individual  tasks  would  thereby  be  more 
tractable  and  make  better  use  of  the  skills  of  individuals  whose 
knowledge  and  expertise  is  likely  to  be  confined  to  a  single  category 
rather  than  the  full  range  of  methods*  This  approach  would  also  allow 
the  work  on  each  subset  of  methods  to  be  performed  concurrently. 

Whatever  the  approach  taken,  producing  a  compendium  of  standard,  usable 
descriptions  of  proven  applied  methods  would  be  an  extremely  valuable 
contribution  to  the  field  of  human  factors  and  consequently  to  the  future 
development  of  human-machine  systems. 


SURVEY  OF  HUMAN  FACTORS  SPECIALISTS  ON  APPLIED  METHODS 

Because  of  the  dearth  of.  Information  on  the  variety  and  use  of  applied 
methods  in  human  factors  work  we  recommend  a  survey  of  human  factors 
practitioners  concerned  with  the  acquisition,  design,  development,  and 
evaluation  or  modification  of  equipment  and  systems.  Such  a  survey  would 
determine  the  Importance  and  frequency  of  U6e  of  existing  applied  methods 
in  their  work;  the  kind  of  information  most  needed  in  human  factors 
applications  for  which  existing  applied  methodologies  are  Inadequate  or 
nonexistent;  and  the  methods  for  which  descriptions  and  guidance  for  use 
are  most  needed. 

The  survey  would  provide  the  necessary  information  on  which  to  base 
documentation,  education,  and  research  efforts.  Review,  codification, 
standardization,  and  documentation  of  existing  methods  should  proceed 
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according  to  the  priorities  of  Importance  and  frequency  of  use  derived 
from  the  survey.  Information  from  the  survey  would  be  useful  In  shaping 
human  factors  curricula  In  colleges  and  universities  so  that  students  can 
be  trained  In  applied  methods  that  they  will  subsequently  need  on  the 
job.  The  continuing  education  needs  of  human  factors  specialists  could 
also  be  met  by  means  of  tutorials  and  symposia  on  the  applied  methods  for 
which  there  the  greatest  need  for  information.  Finally,  the  results 
of  the  survey  would  provide  a  sound  basis  for  basic  research  efforts  to 
extend  or  improve  existing  methods  or  develop  new  methods  to  meet  these 
needs . 

Construction  of  the  survey  instrument  itself  would  require  a  review 
of  the  technical  literature  for  descriptions  and  definitions  of  applied 
methods,  which  the  survey  recipients  would  be  expected  to  rate.  The 
literature  review  would  also  provide  additional  data,  complementary  to 
the  anticipated  survey,  on  the  variety  and  frequency  of  use  of  applied 
methods  reflected  in  the  technical  literature.  A  product  of  this  review 
would  be  a  relatively  comprehensive  bibliography  of  technical  reports  and 
journal  articles  that  discuss  applied  methods  in  more  than  a  cursory 
fashion;  this  bibliographic  information  would  be  extremely  valuable  for 
subsequent  efforts  on  the  codification  and  documentation  of  existing 
methods  and  the  initiation  of  research  efforts  to  extend  these  methods  or 


develop  new  ones 
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EDUCATION  IN  APPLIED  METHODS 


Education  In  Colleges  and  Universities 

The  absence  of  codified  Information  and  the  lack  of  easy  access  to  source 
reports  inhibits  Instruction  In  applied  methods  at  colleges  and 
universities  that  offer  degree  programs  or  courses  in  the  field  of  human 
factors.  General  human  factors  textbooks  give  at  best  only  a  cursory 
overview  of  a  few  applied  methods  and  present  case  study  examples  that 
highlight  the  substantive  Issues  and  results  rather  than  the  methods. 
There  are  no  texts  suitable  either  for  college-level  instruction  or  as  a 
reference  for  practicing  human  factors  specialists  that  adequately  treat 
applied  methods.  The  single  exception.  Research  Techniques  in  Human 
Engineering  (Chapanls,  1959),  discusses  only  a  limited  set  of  methods. 

For  the  most  part,  instructors  must  rely  on  their  own  experience  and  the 
descriptions  of  applied  methods  gleaned  from  the  technical  literature  to 
develop  course  material.  They  have  no  current  and  comprehensive 
reference  works  to  develop  a  balanced  and  thorough  course  in  applied 
methods . 

Human  factors  work  is  diverse  and  Is  performed  in  many 
settings— i.e. ,  military  research  and  development  centers,  other 
government  facilities,  and  commercial  organizations.  Ideally, 

Instruction  in  applied  methods  would  emphasize  the  methods  of  most  use  in 
real-life  settings.  Without  data  on  the  variety  and  frequency  of  use  it 
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is  difficult  to  decide  which  applied  methods  should  be  taught  in  human 
factors  courses  at  the  undergraduate  and  graduate  levels.  Clearly  the 
development  of  a  compendium  of  applied  methods,  as  recommended  in  the 
previous  section,  would  be  of  substantial  benefit  for  formal  educational 
purposes.  Until  such  a  compendium  exists  and  survey  data  is  compiled  on 
the  variety,  frequency  of  use,  and  capabilities  of  applied  methods,  no 
meaningful  recommendations  can  be  made  to  Improve  education  in  applied 
methods  in  colleges  and  universities. 


Continuing  Education  in  Applied  Methods 

Of  equal  concern  is  the  lack  of  suitable  continuing  education  courses  in 
applied  methods  for  practicing  human  factors  specialists. 

The  problem  of  Inadequate  methodological  preparation  in  formal  education 
extends  to  the  work  setting.  At  present  it  appears  that  many  presumably 
well-trained  human  factors  specialists  work  without  adequate  knowledge  of 
applied  methods,  and  what  knowledge  they  do  have  about  these  methods  is 
acquired  on  the  job. 

Curtently  employed  human  factors  specialists  could  benefit  greatly 
from  continuing  education  in  applied  methods  specifically  related  to 
their  current  work.  Development  at  colleges  and  universities  of 
educational  programs  in  applied  methods  that  provide  a  thorough  treatment 
of  a  range  of  applied  methods  would  require  a  substantial  amount  of 
planning  and  course  design  work.  Undoubtedly  the  broad  inception  of 
these  programs,  and  the  realization  of  their  eventual  benefits  in 
practice,  will  be  some  time  in  coming.  Unlike  formal  education  in 
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applied  methods,  however,  the  development  of  courses  for  continuing 
education  could  be  done  more  easily  and  produce  more  Immediate  positive 
effects.  Buman  factors  professionals  are  likely  to  be  more  easily 
educated  because  of  their  general  knowledge  of  human  factors  techniques 
and  the  likelihood  that  they  have  at  least  a  working  familiarity  with 
some  applied  methods.  Because  of  their  previous  education  and 
experience,  continuing  education  courses  for  them  can  be  much  more 
practical,  with  less  emphasis  on  theoretical  foundations.  Based  on  the 
membership  of  the  Human  Factors  Society,  which  numbers  nearly  3,000,  a 
reasonable  estimate  of  the  actual  number  of  practicing  human  factors 
specialists  In  this  country  who  could  benefit  from  continuing  education 
In  applied  methods  is  between  5,000  and  10,000. 

Fostering  and  promoting  continuing  education  by  means  of  tutorials 
on  applied  methods  Is  one  of  the  most  important  and  immediate  ways  to 
Improve  the  field  of  human  factors.  Moreover,  this  kind  of  activity 
could  most  easily  be  initiated  by  military  and  other  federal  agencies 
charged  with  advancing  scientific  and  engineering  knowledge  and 
practice.  These  tutorials  could  directly  benefit  human  factors 
specialists  employed  by  the  government  as  well  as  those  employed  by 
civilian  organizations  that  develop  equipment  and  systems  for  the 
government.  It  is  therefore  recommended  that  initial  tutorials  on 
applied  methods  be  developed  and  conducted  under  the  sponsorship  of  one 
or  more  government  agencies.  While  we  suggest  methods  to  be  discussed  in 
the  tutorial  below,  it  would  be  more  prudent  to  base  the  choice  on  a 
needs  analysis  of  the  data  derived  from  the  survey  recommended  above. 


20 


Such  a  tutorial  could  serve  several  purposes  besides  the  obvious 
one  of  laproving  the  professional  competence  of  human  factors 
specialists.  First,  the  materials  generated  for  the  tutorial  would 
contribute  to  the  development  of  standard  definitions  and  documentation 
of  applied  methods,  since  the  course  materials  would  have  to  describe  the 
subject  methods  with  sufficient  care  and  detail  to  allow  human  factors 
specialists  to  use  them  easily  and  properly.  Second,  the  tutorials  would 
be  a  means  for  validating  a  prior  needs  analysis  of  which  applied  methods 
are  considered  most  important  to  human  factors  practitioners.  Attendance 
at  the  tutorials  would  also  help  answer  a  more  fundamental  question:  Is 
there  genuine  Interest  in  learning  about  applied  methods?  Third,  the 
initial  tutorial  would  serve  as  a  test  to  evaluate  instructional  methods 
and  course  structures  for  training  in  the  use  of  applied  methods. 

It  is  suggested  that  the  Initial  tutorial  should  consist  of  three 
parts:  (1)  an  Introductory  review  of  the  applied  methodologies  within 
each  of  the  five  categories  listed  in  Figure  7-3;  (2)  a  comparison  of 
techniques  within  each  category  and  a  discussion  of  how  to  select  the 
appropriate  method  for  a  particular  application;  and  (3)  detailed 
instruction  and  practical  work  on  a  few  selected  methods.  We  suggest 
five  particular  methodologies  as  subjects  for  the  initial  tutorial: 

Task  analysis; 

Time  line  analysis; 

Activities  analysis; 

Simulation;  and 

Information  Analysis. 
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Because  these  methods  as  well  as  others  are  either  poorly  or 
inconsistently  defined,  brief  definitions  of  the  five  methods  recommended 
for  the  first  tutorial  are  given  In  Appendix  A.  It  would  not  be 
practical  to  cover  more  than  five  methodologies  at  the  Initial  tutorial; 
five  may  even  be  too  many. 

There  are  a  number  of  other  specific  concerns  relevant  to  the  form 
and  development  of  a  tutorial  on  applied  methods.  Experience  has  shown 
tutorials  to  be  only  the  first  step  in  learning  to  use  a  particular 
technique  properly.  Generally,  an  individual  needs  several  days  of 
supervised  application  to  become  competent  in  using  a  particular  method. 
Therefore,  the  tutorial  should  not  be  simply  a  symposium  but  rather 
should  be  a  workshop  in  which  the  attendees  could  gain  hands-on 
experience.  A  by-product  of  the  initial  tutorial  would  be  the 
development  and  testing  of  the  structure  and  effectiveness  of  the  initial 
instructional  methods. 

A  tutorial  on  applied  methods  would  probably  require  10  to  40  hours 
of  planning  and  preparing  for  each  hour  of  instructional  time.  Since  the 
tutorial  should  Include  practical  workshop  exercises  in  addition  to 
lecture,  a  good  part  of -the  effort  of  preparation  would  have  to  be 
devoted  to  development  of  materials.  It  is  likely  that  the  practicum 
would  require  one  or  more  assistants  in  addition  to  the  Instructor. 

An  Individual  or  small  group  should  be  selected  to  develop  a  master 
plan  for  the  tutorial  workshop.  The  primary  goal  would  be  to  choose  the 
methods  to  be  taught  in  the  tutorial.  This  determination  should  be  based 
largely  on  the’ needs  analysis. of  the  data  gathered  from  the  methods 
survey  of  human  factors  practitioners  recommended  abc^e.  The  Individual 
or  group  should  also  address  such  Issues  as  the  number  of  days  the 
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tutorial  should  run,  whether  it  should  be  conducted  independently  or  in 
association  with  a  national  meeting,  the  estimated  costs,  and  the 
selection  of  instructors. 

The  most  obvious  audience  for  the  first  tutorial  are  human  factors 
practitioners,  although  the  needs. of  other  groups  of  professionals  L*»»at 
could  benefit  from  learning  about  applied  methods,  such  as  engineers, 
managers,  students,  and  university  teachers,  should  be  considered  at  some 
point.  Engineers  are  an  Important  audience  since  they  are  likely  to  need 
to  use  applied  methods  in  the  course  of  system  design  and  development  and 
they  are  not  likely  to  know  where  to  seek  information  on  methodologies. 
Managers  are  important  because  of  their  influential  role  in  equipment  and 
system  development.  Due  to  their  position  of  authority,  managers  are 
able  to  Influence  practices  of  their  employees.  College  and  university 
teachers  are  a  relevant  audience,  since  what  they  learn  would  be  passed 
on  to  their  students.  And  students,  especially  students  in  engineering 
and  human  factors,  are  a  particularly  Important  potential  audience 
because  of  their  receptivity  to  new  techniques  and  the  apparent  lack  of 
adequate  education  in  applied  methods  in  colleges  and  universities. 

The  tutorial  format  appropriate  for  human  factors  professionals  may 
not  be  suitable  for  these  other  groups.  If  the  first  tutorial  proves  to 
be  beneficial  to  human  factors  specialists,  it  would  be  worthwhile  to 
design  others  tailored  to  the  backgrounds  and  needs  of  these  other 
groups.  We  recommend  that  tutorials  for  these  other  groups  be  developed 
first  for  engineers  and  subsequently  for  the  remaining  groups. 

For  all  audiences  the  tutorials  should  be  repeated  at  several  times 
and  locations  both  to  make  the  experience  available  to  all  who  are 
interested  and  to  recover  the  initial  development  costs. 


RESEARCH  ON  APPLIED  METHODS 


Each  appllad  method  was  originated  to  fill  seme  particular  need  for 
Information  to  support  system  design,  evaluation,  or  problem  analysis. 
Through  a  succession  of  repeated,  successful  use  In  different,  contexts, 
methods  have  evolved  and  have  become  known  and  accepted  as  tools  of  the 
trade  in  human  factors  work.  Because  they  were  developed  as  a  means  to 
some  practical  end  and  so  vary  in  form  depending  on  the  situations  in 
which  they  are  used,  there  has  never  been  very  much  concern  about  their 
refinement  or  extension.  That  is,  an  applied  method  has  rarely  been 
regarded  as  an  Important  topic  worthy  of  research  Investigation  in  its 
own  right,  independent  of  a  particular  use.  Thlc  lack  of  status  Is 
partly  reflected  and  partly  caused  by  the  absence  of  standard 
documentation  of  applied. methods.  In  addition,  the  people  who  use 
applied  methods  are  practitioners  and,  in  some  sense,  generalists  in 
human  factors  rather  than  specialists  In  methodology.  There  is  no  body 
of  experts  who  devote  their  careers  to  the  study  and  development  of 
applied  methods  rather  than  their  actual  use,  as  there  Is  for 
experimental  design  and  statistical  analysis. 

Applied  methods,  however,  are  the  principal  means  by  which  human 
factors  work  is  accomplished.  In  light  of  their  contibutloo  to  systems 
work,  applied  methods  are  a  sufflcently  Important  topic  to  deserve 
research  attention.  Advances . should  not  depend  solely  on  incidental 
efforts  made  by  human  factors  specialists  In  the  coui.»e  of  their  work. 
Basic  research  specifically  devoted  to  the  validation,  refinement,  and 
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extension  of  existing  methods  and  to  the  development  of  new  methods  is 
essential. 

Improvement  and  Extension  of  Existing  Applied  Methods 

As  previously  discussed,  fundamental  problems  are  the  lack  of  documented 
definitions  and  descriptions  of  existing  applied  methods  and  the  lack  of 
knowledge  about  what  Information  is  needed  in  human  factors  work. 
Documentation  and  survey  work  is  necessary  to  provide  baseline 
descriptions  and  to  help  identify  the  particular  problems  and 
shortcomings  of  existing  methods. 

Without  this  Information  it  is  difficult  to  specify  what  research 
on  which  particular  methods  would  have  the  greatest  value  in  terms  of  its 
contribution  to  the  improvement  of  human  factors  work.  Nonetheless,  we 
propose  some  existing  methods  as  subjects  deserving  research  attention 
because  from  our  experience  it  is  apparent  that  these  methods  are  widely 
used,  critics!  to  system  design  and  development  work,  and  could  be 
substantially  Improved:  workload  analysis;  function  allocation;  task 
analysis;  survey  techniques;  and  protocol  analysis. 

Workload  analysis  is  already  the  subject  of  many  ongoing  research 
programs;  however,  it  is  important  enough  to  merit  expanded  support  for  - 
research  on  workload  assessment  methods.  While  the  five  methods  named 
above  are,  in  our  opinion,  most  deserving  of  research  attention,  the 
order  of  presentation  should  not  be  construed  as  indicating  priorities 
among  them.  There  is  insufficient  knowledge  about  the  needs  of  the  human 
factors  community  to  assign  priorities. 


Development  of  New  Applied  Methods 

In  discussing  current  end  future  problems  and  trends  in  human  factors 
applications  to  system  development,  Meister  (1980,  1982)  has  identified 
those  informational  requirements  of  human  factors  specialists  that  imply 
needs  for  the  development  of  new  applied  methods.  On  the  basis  of  these 
suggestions,  we  make  general  recommendations  for  research  leading  to  the 
development  of  five  new  applied  methods: 

1.  Methods  for  interpreting  or  extrapolating  task/system 
requirements  into  personnel  requirements; 

2.  Performance  measurement  methods  that  express  measures  in 
terms  relative  to  base  rates  for  particular  system 
characteristics  and/or  demands; 

3.  Training  technology  methods  for  translating  task/abilities 
requirements  into  training  programs; 

4.  System  evaluation  methods-- static,  dynamic,  and 
comparative;  and 

5.  Methods  for  describing  and  evaluating  task  or  system  Impact 
on  affective  responses  of  personnel. 
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SUMMARY 

There  Is  a  serious  disparity  between  the  Importance  of  applied 
methodologies  for  human  factors  work,  particularly  systems  and  equipment 
design,  and  the  efforts  being  made  to  document  and  codify  them  In  e 
standard  manner;  to  educate  behavioral  science  and  engineering  students 
in  their  use  in  colleges  and  universities;  to  provide  continuing 
education  In  applied  methods  to  working  human  factors  specialists;  and  to 
engage  in  research  to  improve  existing  applied  methodologies  and  develop 
nev  ones.  It  Is  of  great  Importance  to  document  what  is  currently  known 
about  applied  methods.  Increasing  the  accessibility  of  information  on 
existing  methods  would  be  more  valuable  than  developing  new  methods. 

What  follows  Is  a  summary  of  our  recommendations  with  respect  to  applied 
methods. 

o  Existing  methodologies  should  be  assessed  and  documented  in  a 

i 

codified  compendium  that  provides  standard  descriptions  of  the  most 
useful  applied  methods.  This  compendium  would  serve  both  as  a 
comprehensive  and  readily  available  source  for  learning  about  and  as  a 
basis  for  determining  specific  research  needs. 

o  Human  factors  practitioners  should  be  surveyed  to  determine  the- 
lmportance  and  frequency  of  use  of  existing  applied  methods  in  their 
work;  the  kinds  of  information  most  needed  in  human  factors  applications 
for  which  existing  applied  methods  are  Inadequate  or  nonexistent;  and 
methods  fot  which  they  require  descriptions  and  guidance  for  use. 
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o  Tutorial*  on  applied  methods  should  be  developed  to  meet  the 
continuing  educational  needs  of  human  factors  specialists.  Methods 
recommended  for  the  initial  tutorial  are:  task  analysis;  time  line 
aifalysls;  activities  analysis;  simulation;  and  information  analysis. 

o  Basic  research  should  be  performed  to  Improve  and  extend 
existing  applied  methods.  Methods  in  need  of  research  include:  workload 
analysis;  function  allocation;  task  analysis;  survey  techniques;  and 
protocol  analysis. 

o  Basic  research  Is  also  required  to  develop  new  methods  that  can 
provide  the  Information  needed  by  human  factors  specialists  to  do  their 
work.  New  methods  needed  Include:  (1)  methods  for  interpreting  or 
extrapolating  task/system  requirements  into  personnel  selection 
requirements;  (2)  performance  measurement  methods  that  express  measures 
In  terms  relative  to  base  rates  for  particular  system  characteristics 
and/or  demands;  (3)  training  technology  methods  for  translating 
task/abllltles  requirements  Into  training  programs;  (4)  system  evaluation 
methods~~statlc,  dynamic,  and  comparative;  and  (5)  methods  for  describing 
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and  evaluating  task  or  system  Impact  on  affective  responses  of  personnel. 


APPENDIX  A 


SHORT  DEFINITIONS  OF  APPLIED  METHODS 
RECOMMENDED  AS  SUBJECTS  FOR  TUTORIAL 

Task  Analysis 

Task  analysis  is  the  process  of  analyzing  functional  requirements  of  a 
system  to  ascertain  and  describe  the  tasks  that  people  must  perform. 

Task  analysis  has  two  major  aspects:  The  first  specifies  and  describes 
the  tasks;  the  second  and  more  Important  analyzes  the  specified  tasks  to 
determine  the  number  of  people  needed,  the  skills  and  knowledge  they 
should  have,  and  the  training  necessary.  Results  of  task  analysis  are 
used  in  the  development  of  operating  procedures  and  technical  manuals  and 
the  determination  of  critical  equipment  characteristics  and  task  demands 
imposed  on  people.  The  analytic  method  involves  decomposition  of  task 
content  into  their  constituent  elements,  such  as  stimulus  input,  required 
response,  equipment  output,  and  feedback  information. 

Simulation 

Simulation  is  used  (1)  to  allow  users  to  experience,  in  advance  of  its 
operation,  portions  of  a  system  that  are  more  complex,  more  dangerous,  or 
more  expensive  than  an  experiment  could  allow  for  or  (2)  to  predict 
performance  of  systems  that  do  not  exist.  Simulation  is  a  human  factors 


methodology  only  when  It  Is  combined  with  one  of  the  observational  or 
measurement  methodologies.  And  to  extrapolate  the  observations  or 
measurements  to  the  real  world  requires  a  determination  of  the  extent  to 
which  things  that  affect  the  observations  of  Interest  are  realistically 
portrayed  in  the  simulation.  Bov  to  make  this  determination 
(cost/ transfer  function,  part  versus  whole  task  simulation,  which  things 
to  simulate)  Is  the  key  part  of  the  technology  that  is  still  largely 
unresolved.  In  the  absence  of  other  effective  means  of  predicting  the 
behavioral  consequences  of  system  design,  simulation  is  crucial. 

Time  Line  Analysis 

Time  line  analysis  organizes  a  detailed  task  list  for  the  operational 
scenario  and  procedures  Into  serial  order  and  plots  the  times  of 
Individual  tasks  in  sequence  against  a  time  base.  It  portrays 
sequential,  parallel,  .repeated,  and/or  Intermittent  tasks  according  to 
what  Is  done.  The  resulting  accumulation  of  tasks  and  total  performance 
time  can  be  used  to  appraise: 

l 

1.  The  validity  of  the  operations  to  be  performed  in  contributing 
to  system  objectives; 

2.  The  feasibility  of  performing  required  tasks  within  the  required 
time; 

3.  Antecedent  hardware  and  operations  conditions  to  ensure  that  the 
requirements  of  each  task  element  are  met; 
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4.  The  compatibility  of  demands  on  the  operator,  ensuring  that 
antecedent  tasks  are  identified  and  performed,  required  skills 
and  performances  are  feasible  and  practical,  and  difficult, 
complex,  or  conflicting  demands  are  avoided;  and 

5.  Workload  demands,  by  comparing  time  requirements  to  complete  a 
task  series  to  the  time  available  for  completion  within  the 
constraints  of  a  given  system. 

Information  Analysis 

Information  analysis  identifies  Information  and  Its  flow  through  a 
system,  usually  as  perceived  from  a  user's  viewpoint.  For  example,  the 
flow  of  information  necessary  for  the  operation  of  an  office  differs  from 
the  flow  of  documents  through  that  office.  Certain  system  actions  occur 
to  the  Information  received,  which  in  turn  becomes  inputs  to  subsequent 
actions.  Information  analyses  enable  human  factors  specialists  to  assess 
and  design  the  Information  requirements  of  the  user  interfaces. 

Activity  Analysis 

In  many  situations  involving  field  environments,  simulations,  or 
mock-ups,  it  is  desirable  and  useful  to  catalog  the  distribution  and/or 
sequential  dependencies  of  workers’  activities.  In  activity  analysis  an 
observer  periodically  or  aperiodically  samples  the  work  being  performed 
and  classifies  the  results  into  a  set  of  categories.  The  data  may  be 
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obtained  from  direct  observation  or  from  video  or  film  recording. 

Individual  samples  are  then  aggregated  into  activity  frequency  tables  or 
* 

graphs  or  state  transition  diagrams.  These  analyses  are  especially 
useful  for  documenting  the  way  in  which  task  requirements  change  with 
alternative  system  designs  or  environments  or  for  estimates  of  relative 
cost  effectiveness,  manning  requirements,  or  simply  for  understanding  how 
individuals  or  groups  spend  their  time. 


REFERENCES 


Chapania,  , 
1959 

Geer,  C.  W. 
1981 


Melster,  D. 
1980 


Melster,  D. 
1982 


Baltimore,  m. 

John*  Hopkins  University  Press, 

agaajjnglneerto,  Fror.,.,,...  o..,^  ^ 


AFAMRL-TR-81-35. 


Wright-Patteraon  Air  Force  Base, 


Air  Force  Aero.pece  Medical  He.earch  Laboratory. 


Ohio: 


Human  Factor,  for  the  Futur.~Tr.ng.  .nd  Speculation.,  m 

tssssiiuLsiUatsis^B.  Bwn  F,ct„. 

Karl. tad t,  Sveden. 

National  Defense  Research  Institute. 


The  role  of  human  factors  In  sve«-An  j  , 

actors  m  system  development.  Applied 

Ergonoml  cs  13 :  .17  9-1?/; 


33 


Parks,  D.  L. 
1976 


Topol  Her, 
1981 


Wllllgea,  R. 
1980 


,  and  Springer,  W.  E. 

Human  Factors  Engineering  Analytic  Process  Definition  and 
Criterion  Development  for  CAFES  (Computer  Aided 
Function-Allocation  Evaluation  System)*  Warminster,  Pa.: 
Naval  Air  Development  Center. 

.  A. 

Methods:  past  approaches,  current  trends  and  future 
requirements.  In  M.  J.  Moraal  and  K.  F.  Kralss,  eds., 
Manned  System  Design.  New  Tork:  Plenum  Press. 

C.,  and  Topmiller,  D*  A. 

Task  III;  Technology  Assessment  of  Human  Factors 
Engineering  In  the  Air  Force.  Vrlght-Patterson  Air  Force 
Base,  (Milo:  Air  Force  Systems  Command. 


062083 


